BayLing-Duplex: 单一自回归LLM的原生全双工语音对话
BayLing-Duplex: Native Full-Duplex Speech Dialogue with a Single Autoregressive LLM
AI总结：
提出BayLing-Duplex，一种原生全双工语音语言模型，通过单个自回归LLM决定何时听、说和停止，无需外部VAD模块，仅用少量特殊标记实现，在少量微调数据上达到高交互成功率并提升响应质量。
摘要：
实时全双工语音交互是下一代语音聊天机器人的关键特性，允许模型同时听和说，并处理重叠、犹豫和插话等自然现象。现有的语音语言模型（如LLaMA-Omni和GLM-4-Voice）仍然是基于回合的，并依赖外部语音活动检测（VAD）模块来标记用户回合的结束，这从根本上限制了它们的交互能力。在本文中，我们介绍了BayLing-Duplex，一种原生全双工SpeechLM，其中单个自回归LLM决定何时听、何时说以及何时停止，无需辅助的回合切换模块。该设计仅在标准词汇表中添加少量特殊标记，因此可以跨LLM迁移，并重用现有的训练和服务堆栈，无需架构适配。从公开的GLM-4-Voice检查点开始，仅使用400K全双工样本进行微调，随后进行轻量级DPO阶段，BayLing-Duplex在InstructS2S-Eval上达到92%的回合切换成功率和100%的打断成功率，同时将语音响应分数从Moshi的2.17提升到3.39。BayLing-Duplex在Llama Questions、Web Questions和Alpaca-Eval上也达到或超过了其基于回合的对应版本，表明同时听和说建模不会牺牲响应质量。
英文摘要：
Real-time, full-duplex speech interaction is a key feature of next-generation spoken chatbots, allowing the model to listen and speak at the same time and to handle natural phenomena such as overlap, hesitation, and barge-in. Existing speech language models (SpeechLMs) such as LLaMA-Omni and GLM-4-Voice are still turn-based and rely on an external Voice Activity Detection (VAD) module to mark the end of the user's turn, which fundamentally limits their interactive ability. In this paper, we introduce BayLing-Duplex, a native full-duplex SpeechLM where a single autoregressive LLM decides when to listen, when to speak, and when to stop, with no auxiliary turn-taking module. The design adds only a few special tokens to the standard vocabulary, so it transfers across LLMs and reuses existing training and serving stacks with no architectural adaptation. Starting from the public GLM-4-Voice checkpoint and using only 400K full-duplex samples for fine-tuning followed by a lightweight DPO stage, BayLing-Duplex reaches 92% turn-taking success and 100% interruption success on InstructS2S-Eval, while improving the speech-response score from 2.17 to 3.39 over Moshi. BayLing-Duplex also matches or surpasses its turn-based counterpart on Llama Questions, Web Questions, and Alpaca-Eval, showing that simultaneous listen-and-speak modeling does not sacrifice response quality.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
