如何连接语音基础模型与大语言模型？什么重要，什么不重要
How to Connect Speech Foundation Models and Large Language Models? What Matters and What Does Not
AI总结：
本文通过组合5种适配器、2种LLM和2种SFM，在自动语音识别和语音翻译任务上系统评估语音基础模型与大语言模型连接方案，发现SFM对下游性能影响最大，适配器影响中等且依赖具体模型选择。
摘要：
大语言模型（LLM）取得的卓越性能推动了研究者将其用于广泛任务和输入模态的努力。在语音转文本（S2T）任务中，新兴的解决方案是通过适配器模块将语音基础模型（SFM）编码器的输出投影到LLM嵌入空间中。然而，目前尚无工作研究下游任务性能在多大程度上依赖于每个组件（SFM、适配器、LLM），也没有研究适配器的最佳设计是否取决于所选的SFM和LLM。为填补这一空白，我们在两个广泛使用的S2T任务（即自动语音识别和语音翻译）上评估了5种适配器模块、2种LLM（Mistral和Llama）以及2种SFM（Whisper和SeamlessM4T）的组合。我们的结果表明，SFM在下游性能中起着关键作用，而适配器的选择影响中等，并且取决于SFM和LLM。
英文摘要：
The remarkable performance achieved by Large Language Models (LLM) has driven research efforts to leverage them for a wide range of tasks and input modalities. In speech-to-text (S2T) tasks, the emerging solution consists of projecting the output of the encoder of a Speech Foundational Model (SFM) into the LLM embedding space through an adapter module. However, no work has yet investigated how much the downstream-task performance depends on each component (SFM, adapter, LLM) nor whether the best design of the adapter depends on the chosen SFM and LLM. To fill this gap, we evaluate the combination of 5 adapter modules, 2 LLMs (Mistral and Llama), and 2 SFMs (Whisper and SeamlessM4T) on two widespread S2T tasks, namely Automatic Speech Recognition and Speech Translation. Our results demonstrate that the SFM plays a pivotal role in downstream performance, while the adapter choice has moderate impact and depends on the SFM and LLM.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
