通过半隐式跨语言思维链推理增强语音大语言模型中的非核心语言指令遵循
Enhancing Non-Core Language Instruction-Following in Speech LLMs via Semi-Implicit Cross-Lingual CoT Reasoning
AI总结：
针对语音大语言模型在非核心语言指令遵循上的不足，提出半隐式跨语言语音思维链框架XS-CoT，通过整合语音到文本翻译与推理token压缩，在仅用少量训练数据下显著提升非核心语言响应质量并大幅降低推理延迟。
摘要：
大语言模型已被扩展至语音领域，促成了语音大语言模型（SLLMs）的发展。尽管现有的SLLMs在核心语言（如英语）的语音指令遵循方面表现出色，但由于配对的语音-文本数据稀缺以及多语言语义推理能力有限，它们在处理非核心语言时往往面临困难。为解决这一问题，我们提出了半隐式跨语言语音思维链（XS-CoT）框架，将语音到文本的翻译整合到SLLMs的推理过程中。XS-CoT生成四种类型的token：核心语言和非核心语言的指令与响应token，从而实现推理能力的跨语言迁移。为减少生成目标非核心响应token时的推理延迟，我们在XS-CoT中引入半隐式CoT方案，在训练期间逐步压缩前三类中间推理token，同时保留全局推理逻辑。借助核心语言强大的推理能力，与在两个代表性SLLMs（Qwen2-Audio和SALMONN）上进行直接监督微调相比，XS-CoT使非核心语言的响应在GPT-4评分上最高提升45%。此外，半隐式XS-CoT将token延迟降低50%以上，且GPT-4评分仅有轻微下降。重要的是，通过利用核心语言的推理能力，XS-CoT仅需少量高质量的非核心语言训练数据。为支持训练，我们还开发了数据管道，并开源了日语、德语和法语的语音指令遵循数据集。
英文摘要：
Large language models have been extended to the speech domain, leading to the development of speech large language models (SLLMs). While existing SLLMs demonstrate strong performance in speech instruction-following for core languages (e.g., English), they often struggle with non-core languages due to the scarcity of paired speech-text data and limited multilingual semantic reasoning capabilities. To address this, we propose the semi-implicit Cross-lingual Speech Chain-of-Thought (XS-CoT) framework, which integrates speech-to-text translation into the reasoning process of SLLMs. The XS-CoT generates four types of tokens: instruction and response tokens in both core and non-core languages, enabling cross-lingual transfer of reasoning capabilities. To mitigate inference latency in generating target non-core response tokens, we incorporate a semi-implicit CoT scheme into XS-CoT, which progressively compresses the first three types of intermediate reasoning tokens while retaining global reasoning logic during training. By leveraging the robust reasoning capabilities of the core language, XS-CoT improves responses for non-core languages by up to 45\% in GPT-4 score when compared to direct supervised fine-tuning on two representative SLLMs, Qwen2-Audio and SALMONN. Moreover, the semi-implicit XS-CoT reduces token delay by more than 50\% with a slight drop in GPT-4 scores. Importantly, XS-CoT requires only a small amount of high-quality training data for non-core languages by leveraging the reasoning capabilities of core languages. To support training, we also develop a data pipeline and open-source speech instruction-following datasets in Japanese, German, and French.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
