Freeze-Omni：一种基于冻结大语言模型的智能低延迟语音到语音对话模型
Freeze-Omni: A Smart and Low Latency Speech-to-speech Dialogue Model with Frozen LLM
AI总结：
本研究提出冻结LLM参数的Freeze-Omni语音-文本多模态架构，通过三阶段训练、多任务训练实现低延迟双向语音对话，保留骨干LLM智能水平，避免灾难性遗忘。
摘要：
快速发展的大语言模型（LLM）催生了大量智能应用，尤其是GPT-4o出色的双向语音交互能力为用户带来了令人印象深刻的体验。近期研究人员提出了多个该方向的多模态大语言模型，可实现用户与智能体之间的语音到语音对话。本文提出一种名为Freeze-Omni的新型语音-文本多模态大语言模型架构。我们的核心贡献是，在整个训练过程中保持文本大语言模型参数冻结的前提下，可轻松将语音输入与输出模态接入该模型。我们设计了三阶段训练策略对语音输入与输出进行建模，使Freeze-Omni仅需利用文本-语音配对数据（如ASR和TTS数据）以及6万条多轮文本问答数据，在8块GPU上即可获得语音到语音对话能力。此外，我们能有效保障Freeze-Omni在语音模态下的智能水平与其骨干大语言模型的文本模态智能水平相当，同时实现低延迟的端到端语音响应。我们还设计了一种通过多任务训练实现双向对话能力的方法，让Freeze-Omni具备更自然的用户与智能体对话风格。综上，Freeze-Omni在大语言模型冻结的条件下，具备开展基于多模态大语言模型的语音到语音对话的巨大潜力，可避免有限数据与训练资源导致的灾难性遗忘问题。
英文摘要：
Rapidly developing large language models (LLMs) have brought tremendous intelligent applications. Especially, the GPT-4o's excellent duplex speech interaction ability has brought impressive experience to users. Researchers have recently proposed several multi-modal LLMs in this direction that can achieve user-agent speech-to-speech conversations. This paper proposes a novel speech-text multimodal LLM architecture called Freeze-Omni. Our main contribution is that the speech input and output modalities can be easily connected to a textual LLM while keeping the LLM's parameters frozen throughout the training process. We design a three-stage training strategy for modeling both the speech input and output, enabling Freeze-Omni to obtain speech-to-speech conversation ability using text-speech paired data (such as ASR and TTS data) and only 60,000 multi-round text Q&A data on 8 GPUs. Moreover, we can effectively ensure that the intelligence of the Freeze-Omni in the speech modality is at the same level compared with that in the text modality of its backbone LLM, while achieving low latency end-to-end spoken response. In addition, we also designed a method to achieve duplex dialogue ability through multi-task training, giving Freeze-Omni a more natural style of dialogue ability between users and agents. In summary, Freeze-Omni holds great potential to conduct speech-to-speech dialogue based on a multimodal LLM under the condition of a frozen LLM, avoiding the catastrophic forgetting problem caused by limited data and training resources.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
