Mini-Omni：语言模型可以在流式思考的同时听和说
Mini-Omni: Language Models Can Hear, Talk While Thinking in Streaming
AI总结：
本文提出Mini-Omni，首个开源端到端实时语音交互模型，通过文本指导语音生成和批量并行推理实现流式对话，并引入VoiceAssistant-400K数据集，以最小退化保留语言能力。
摘要：
近年来，语言模型的进展取得了显著成果。GPT-4o作为一个新的里程碑，实现了与人类的实时对话，展现出接近人类的自然流畅度。这种人机交互要求模型具备直接以音频模态进行推理并以流式方式生成输出的能力。然而，这仍然超出了当前学术界模型的能力范围，因为它们通常依赖额外的TTS系统进行语音合成，导致不理想的延迟。本文介绍了Mini-Omni，一个基于音频的端到端对话模型，能够实现实时语音交互。为了实现这一能力，我们提出了一种文本指导的语音生成方法，并在推理过程中采用批量并行策略以进一步提升性能。我们的方法还有助于以最小的退化保留原始模型的语言能力，使其他工作能够建立实时交互能力。我们将这种训练方法称为“Any Model Can Talk”。我们还引入了VoiceAssistant-400K数据集，用于微调优化语音输出的模型。据我们所知，Mini-Omni是首个完全端到端、开源的实时语音交互模型，为未来研究提供了宝贵的潜力。
英文摘要：
Recent advances in language models have achieved significant progress. GPT-4o, as a new milestone, has enabled real-time conversations with humans, demonstrating near-human natural fluency. Such human-computer interaction necessitates models with the capability to perform reasoning directly with the audio modality and generate output in streaming. However, this remains beyond the reach of current academic models, as they typically depend on extra TTS systems for speech synthesis, resulting in undesirable latency. This paper introduces the Mini-Omni, an audio-based end-to-end conversational model, capable of real-time speech interaction. To achieve this capability, we propose a text-instructed speech generation method, along with batch-parallel strategies during inference to further boost the performance. Our method also helps to retain the original model's language capabilities with minimal degradation, enabling other works to establish real-time interaction capabilities. We call this training method "Any Model Can Talk". We also introduce the VoiceAssistant-400K dataset to fine-tune models optimized for speech output. To our best knowledge, Mini-Omni is the first fully end-to-end, open-source model for real-time speech interaction, offering valuable potential for future research.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
