AzeroS：通过自生成指令-free微调扩展LLM到语音
AzeroS: Extending LLM to Speech with Self-Generated Instruction-Free Tuning
AI总结：
AZeroS通过自生成指令-free微调方法，无需任务特定数据即可实现语音领域LLM的高效扩展，取得最佳泛化性能。
摘要：
将大型语言模型（LLM）扩展到语音领域最近引起了广泛关注。一种典型方法是通过投影模块将预训练的LLM与音频编码器连接起来，并在大规模、任务特定的指令微调数据集上训练所得模型。然而，为特定需求收集此类指令微调数据是耗时的，而且通过这种方式训练的模型在面对未见过的任务时泛化能力较差。在本工作中，我们首先提出，当语音-LLM通过自生成指令-free微调（SIFT）进行训练时，其最强的泛化能力得以实现，其中监督信号是通过冻结的LLM使用语音的文本表示作为输入生成的。我们提出的SIFT范式消除了收集任务特定问题-答案对的需要，并在理论上实现了对未见任务的最佳泛化。基于此范式，我们引入了AZeroS（Auden零指令微调语音-LLM），该模型在公开可用的语料库中训练，包括约25000小时的语音及其ASR转录文本和3000小时的语音及其副语言标签。基于Qwen2.5-7B-Instruct，模型仅更新了两个轻量级投影模块（每个模块有2380万个参数），同时保持LLM和音频编码器冻结。尽管训练成本极低且数据规模有限，AZeroS在语义和副语言基准测试中均取得了最先进的性能，包括VoiceBench、AIR-Bench Foundation（Speech）和AIR-Bench Chat（Speech）。\n
英文摘要：
Extending large language models (LLMs) to the speech domain has recently gained significant attention. A typical approach connects a pretrained LLM with an audio encoder through a projection module and trains the resulting model on large-scale, task-specific instruction-tuning datasets. However, curating such instruction-tuning data for specific requirements is time-consuming, and models trained in this manner often generalize poorly to unseen tasks. In this work, we first formulate that the strongest generalization of a speech-LLM is achieved when it is trained with Self-Generated Instruction-Free Tuning (SIFT), in which supervision signals are generated by a frozen LLM using textual representations of speech as input. Our proposed SIFT paradigm eliminates the need for collecting task-specific question-answer pairs and yields the theoretically best generalization to unseen tasks. Building upon this paradigm, we introduce AZeroS (Auden Zero-instruction-tuned Speech-LLM), which is trained on speech-text pairs derived from publicly available corpora, including approximately 25,000 hours of speech with ASR transcripts and 3,000 hours of speech with paralinguistic labels. Built upon Qwen2.5-7B-Instruct, the model updates only two lightweight projection modules (23.8 million parameters each), while keeping both the LLM and audio encoders frozen. Despite the minimal training cost and modest data scale, AZeroS achieves state-of-the-art performance on both semantic and paralinguistic benchmarks, including VoiceBench, AIR-Bench Foundation (Speech), and AIR-Bench Chat (Speech).
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
