Fun-Audio-Chat 技术报告
Fun-Audio-Chat Technical Report
AI总结：
Fun-Audio-Chat通过双分辨率语音表示和核心鸡尾酒训练，解决语音-文本模型的时间分辨率不匹配和灾难性遗忘问题，实现高效且高质量的音频理解与生成。
摘要：
近年来，联合语音-文本模型在实现无缝语音交互方面展现出巨大潜力。然而，现有模型面临关键挑战：语音标记（25Hz）与文本标记（约3Hz）之间的时间分辨率不匹配，会稀释语义信息，导致计算成本高，并引起文本LLM知识的灾难性遗忘。我们引入了Fun-Audio-Chat，这是一种大型音频语言模型，通过我们之前工作的DrVoice中的两项创新来解决这些限制。首先，双分辨率语音表示（DRSR）：共享的LLM以高效的5Hz（通过标记分组）处理音频，而语音精炼头部则在25Hz下生成高质量的标记，平衡了效率（约50%的GPU减少）和质量。其次，核心鸡尾酒训练，一种两阶段微调，带有中间合并，可以缓解灾难性遗忘。我们随后应用多任务DPO训练来增强鲁棒性、音频理解、指令遵循和语音共情能力。这种多阶段后训练使Fun-Audio-Chat能够保留文本LLM知识，同时获得强大的音频理解、推理和生成能力。与最近的LALMs需要大规模音频-文本预训练不同，Fun-Audio-Chat利用预训练模型和大量后训练。Fun-Audio-Chat 8B和MoE 30B-A3B在语音到文本和语音到语音任务上取得了竞争性表现，在口语问答基准测试中排名同类模型前列。它们还在音频理解、语音功能调用、指令遵循和语音共情方面取得了竞争性或更优异的表现。我们开发了Fun-Audio-Chat-Duplex，一种全双工变种，在口语问答和全双工交互上表现强劲。我们开源了Fun-Audio-Chat-8B，提供了训练和推理代码，并提供了一个交互式演示，网址为https://github.com/FunAudioLLM/Fun-Audio-Chat。
英文摘要：
Recent advancements in joint speech-text models show great potential for seamless voice interactions. However, existing models face critical challenges: temporal resolution mismatch between speech tokens (25Hz) and text tokens (~3Hz) dilutes semantic information, incurs high computational costs, and causes catastrophic forgetting of text LLM knowledge. We introduce Fun-Audio-Chat, a Large Audio Language Model addressing these limitations via two innovations from our previous work DrVoice. First, Dual-Resolution Speech Representations (DRSR): the Shared LLM processes audio at efficient 5Hz (via token grouping), while the Speech Refined Head generates high-quality tokens at 25Hz, balancing efficiency (~50% GPU reduction) and quality. Second, Core-Cocktail Training, a two-stage fine-tuning with intermediate merging that mitigates catastrophic forgetting. We then apply Multi-Task DPO Training to enhance robustness, audio understanding, instruction-following and voice empathy. This multi-stage post-training enables Fun-Audio-Chat to retain text LLM knowledge while gaining powerful audio understanding, reasoning, and generation. Unlike recent LALMs requiring large-scale audio-text pre-training, Fun-Audio-Chat leverages pre-trained models and extensive post-training. Fun-Audio-Chat 8B and MoE 30B-A3B achieve competitive performance on Speech-to-Text and Speech-to-Speech tasks, ranking top among similar-scale models on Spoken QA benchmarks. They also achieve competitive to superior performance on Audio Understanding, Speech Function Calling, Instruction-Following and Voice Empathy. We develop Fun-Audio-Chat-Duplex, a full-duplex variant with strong performance on Spoken QA and full-duplex interactions. We open-source Fun-Audio-Chat-8B with training and inference code, and provide an interactive demo, at https://github.com/FunAudioLLM/Fun-Audio-Chat .
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
