解锁语音-文本组合能力：无需指令微调的指令跟随语音语言模型
Unlocking Speech-Text Compositional Powers: Instruction-Following Speech Language Models without Instruction Tuning
AI总结：
提出SpeechCombine，通过单轮30k小时语音预训练和权重组合，无需指令微调即可实现指令跟随，有效将文本LLM能力迁移至语音域。
摘要：
语音语言模型（SLM）的指令微调比基于文本的大语言模型（LLM）更具挑战性，因为它需要学习新的模态以及文本LLM支持的指令之外的广泛语音特定指令。现有的SLM训练方法主要通过合成大规模语音预训练和指令微调数据集来复制文本LLM训练范式。然而，这种策略难以扩展，因为语音序列比文本序列长得多。在本文中，我们提出SpeechCombine，一种无需任何指令微调、仅使用30k小时数据的单轮语音预训练训练的指令跟随语音语言模型。从文本LLM基座模型开始，我们对语音话语进行连续预训练以获得语音适应模型，然后直接将其权重与文本LLM的指令微调版本和基座版本之间的权重差相结合。我们的结果表明，这种简单的组合策略不仅保留了原始文本LLM的知识和能力，而且有效地将其迁移到语音域。这些发现为SLM训练提供了一条新方向，避免了对大规模语音数据的依赖。
英文摘要：
Instruction tuning for speech language models (SLMs) is substantially more challenging than for text-based large language models (LLMs), as it requires learning a new modality and a wide range of speech-specific instructions in addition to those supported by text LLMs. Existing SLM training approaches largely replicate the text LLM training paradigm by synthesizing large-scale speech pre-training and instruction-tuning datasets. However, this strategy is difficult to scale, since speech sequences are significantly longer than text sequences. In this paper, we propose SpeechCombine, an instruction-following speech language model trained without any instruction tuning, using only a single round of speech pre-training on 30k hours of data. Starting from a text LLM base model, we perform continuous pre-training on speech utterances to obtain a speech-adapted model, and then directly combine its weights with the weight difference between the instruction-tuned and base versions of the text LLM. Our results show that this simple combination strategy not only preserves the knowledge and capabilities of the original text LLM, but also effectively transfers them to the speech domain. These findings suggest a new direction for SLM training that avoids reliance on massive speech data.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
