arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.17124cs.CLcs.AIeess.AS
收藏

D2:基于半双工主干的高能力全双工口语语言模型

D2: High-Capability Full-Duplex Spoken Language Models from Half-Duplex Backbones

MIT CSAIL(麻省理工学院计算机科学与人工智能实验室);MIT-IBM Watson AI Lab(麻省理工学院-IBM沃森人工智能实验室)

Cheng-Kuang Chang, Alexander H. Liu, Brian Kingsbury, Hilde Kuehne, Rogerio Feris, Samuel Thomas, James Glass, Kai-Wei Chang

AI总结:

针对FD-SLM通用能力弱于HD-SLM的问题,提出D2框架将预训练HD-SLM转换为高能力FD-SLM,在Qwen3-Omni等模型上取得优异性能。

摘要:

全双工口语语言模型(FD-SLM)支持同时听和说,但在通用能力上常落后于半双工口语语言模型(HD-SLM)。我们提出对话到双工(Dialogue-to-Duplex,D2)框架,该框架以精细时间粒度将预训练HD-SLM转换为高能力FD-SLM:通过无解码器表示蒸馏使原始音频编码器适配流式处理,通过监督微调学习全双工交互。我们系统研究了块大小对全双工交互、通用能力和推理计算的影响,将D2应用于Qwen3-Omni和LLaMA-Omni 2两种不同的HD-SLM架构。在12.5Hz更新率下,Qwen3-Omni-D2实现99.2%的轮次切换成功率、521毫秒延迟、99.5%的打断成功率,以及70.1的SOTA VoiceBench分数。

英文摘要:

Full-duplex spoken language models (FD-SLMs) support simultaneous listening and speaking, but often lag behind half-duplex spoken language models (HD-SLMs) in general capabilities. We propose Dialogue-to-Duplex (D2), a framework that converts pretrained HD-SLMs into high-capability FD-SLMs at fine temporal granularity. D2 adapts the original audio encoder to streaming through decoder-free representation distillation and learns full-duplex interaction through supervised fine-tuning. We systematically study the effects of chunk size on full-duplex interaction, general capabilities, and inference computation. We apply D2 to two distinct HD-SLM architectures, Qwen3-Omni and LLaMA-Omni 2. At a 12.5Hz update rate, Qwen3-Omni-D2 achieves 99.2% turn-taking success with 521-ms latency, 99.5% interruption success, and a state-of-the-art VoiceBench score of 70.1.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑