CharDuplex：构建角色一致的全双工口语对话模型

中文摘要 AI 辅助

全双工语音模型正在将语音交互推向超越传统轮流对话的模式，然而自然对话不仅取决于智能体何时说话，还取决于它作为对话角色如何表现。我们提出了CharDuplex，一种角色驱动的全双工语音模型，将实时口语交互与角色条件化行为相结合。我们首先将GLM-4-Voice适配为始终在线的双流架构，并训练模型进行全双工对话。然后，一个全自动流水线从开源角色描述中构建角色条件化对话数据，用于角色条件化监督微调。该模型进一步通过所提出的FDGym进行优化，其中由LLM模拟的用户与模型动态交互，使得模型能够在不断演化的多轮交互上进行强化学习。在SpeechRole-Eval基准上，CharDuplex在评估的开源模型中取得了最高的平均得分，同时与闭源系统保持竞争力。它还展示了具有竞争力的通用语音智能和强大的全双工交互能力。CharDuplex展示了一种实用的训练方案，用于构建不仅具有交互性而且角色一致的全双工语音助手。

英文摘要

Full-duplex speech models are moving voice interaction beyond conventional turn-taking, yet natural conversation is shaped not only by when an agent speaks, but also by how it behaves as a conversational character. We present CharDuplex, a character-driven full-duplex speech model that combines real-time spoken interaction with persona-conditioned behavior. We first adapt GLM-4-Voice to an always-on dual-stream architecture and train the model for full-duplex conversation. Then a fully automated pipeline constructs character-conditioned dialogue data from open-source character descriptions for character-conditioned supervised fine-tuning. The model is further refined with the proposed FDGym, where an LLM-simulated user dynamically interacts with the model, enabling reinforcement learning over evolving multi-turn interactions. On SpeechRole-Eval, CharDuplex achieves the highest average score among the evaluated open-source models, while remaining competitive with closed-source systems. It also demonstrates competitive general speech intelligence and strong full-duplex interaction capabilities. CharDuplex demonstrates a practical training recipe for building full-duplex voice assistants that are not only interactive, but also character-consistent.

发表机构

• Nanyang Technological University（南洋理工大学）
• AI Singapore（新加坡人工智能研究院）
• University of California, Berkeley（加州大学伯克利分校）

机构由 AI 辅助整理，请以论文原文为准。