SLM：弥合语音与文本基础模型之间的细微差距
SLM: Bridge the thin gap between speech and text foundation models
AI总结：
本研究提出联合语音与语言模型SLM，通过仅训练1%参数量的适配器连接预训练语音和文本基础模型，既在传统语音任务上表现优异，又实现了零样本指令跟随的多样生成能力，验证了两类模型表征差距可通过简单适配弥合。
摘要：
我们提出了一种联合语音与语言模型（Speech and Language Model，简称SLM），这是一种多任务、多语言的双模态模型，充分利用了预训练的语音和语言基础模型的能力。SLM冻结预训练基础模型以最大程度保留其原有能力，仅训练一个简单适配器，其参数量仅为基础模型的1%（1.56亿）。\n这种适配不仅使SLM在语音识别（ASR）、语音翻译（AST）等传统任务上取得优异性能，还赋予了其零样本指令跟随的全新能力，可完成更多样化的任务：给定语音输入和文本指令，SLM能够执行未见过的生成任务，包括利用实时上下文的上下文偏置ASR、对话生成、语音续写以及问答等。\n我们的方法表明，预训练语音模型与语言模型之间的表征差距可能比人们预想的更小，通过简单的适配机制即可弥合这一差距。因此，SLM不仅训练高效，还继承了不同模态基础模型已具备的强大能力。
英文摘要：
We present a joint Speech and Language Model (SLM), a multitask, multilingual, and dual-modal model that takes advantage of pretrained foundational speech and language models. SLM freezes the pretrained foundation models to maximally preserves their capabilities, and only trains a simple adapter with just 1\% (156M) of the foundation models' parameters. This adaptation not only leads SLM to achieve strong performance on conventional tasks such as speech recognition (ASR) and speech translation (AST), but also introduces the novel capability of zero-shot instruction-following for more diverse tasks: given a speech input and a text instruction, SLM is able to perform unseen generation tasks including contextual biasing ASR using real-time context, dialog generation, speech continuation, and question answering, etc. Our approach demonstrates that the representational gap between pretrained speech and language models might be narrower than one would expect, and can be bridged by a simple adaptation mechanism. As a result, SLM is not only efficient to train, but also inherits strong capabilities already acquired in foundation models of different modalities.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
