TurnGuide: 通过动态轮次级文本-语音交错增强有意义的全双工口语交互
TurnGuide: Enhancing Meaningful Full Duplex Spoken Interactions via Dynamic Turn-Level Text-Speech Interleaving
AI总结：
提出TurnGuide方法，通过动态分割助手语音为对话轮次并交错生成轮次级文本和语音，解决全双工语音语言模型在连续双通道音频中集成离散文本令牌导致的时间对齐问题，显著提升语义连贯性和轮次交互性能。
摘要：
全双工语音语言模型（FD-SLMs）是专门的基础模型，旨在通过建模复杂的对话轮次（如打断、反馈和重叠语音）来实现自然的实时口语交互。端到端（e2e）FD-SLMs利用真实世界的双通道对话数据捕捉细微的双说话者对话模式以实现类人交互，但由于语音序列过长和高质量口语对话数据有限，其对话能力往往比纯文本对话有所下降。尽管交错文本-语音生成可以缓解这种退化，但将离散文本令牌集成到连续双通道音频流中可能会破坏流畅交互所需的时间对齐。为了解决这个问题，我们提出了TurnGuide，一种用于e2e FD-SLMs的新型文本-语音交错生成方法，该方法动态地将助手语音分割成对话轮次，并交错生成轮次级文本和语音。这种方法使FD-SLMs能够整合LLMs的语义智能，同时不损害自然的声学流畅性。大量实验表明，TurnGuide不仅显著提升了e2e FD-SLMs生成语义有意义且连贯语音的能力，而且在各种轮次事件上达到了最先进的性能。演示请访问此https URL。代码请访问此https URL。
英文摘要：
Full-Duplex Speech Language Models (FD-SLMs) are specialized foundation models designed to enable natural, real-time spoken interactions by modeling complex conversational turn-taking such as interruptions, backchannels, and overlapping speech. End-to-end (e2e) FD-SLMs leverage real-world double-channel conversational data to capture nuanced two-speaker dialogue patterns for human-like interactions, but their conversational abilities often degrade compared to pure-text conversation due to prolonged speech sequences and limited high-quality spoken dialogue data. Although interleaved text-speech generation could mitigate this degradation, integrating discrete text tokens into continuous double-channel audio streams could disrupt the precise time alignment required for fluid interaction. To address this, we propose TurnGuide, a novel text-speech interleaved generation approach for e2e FD-SLMs that dynamically segments assistant speech into dialogue turns and interleaves turn-level text and speech generation. This approach allows FD-SLMs to integrate the semantic intelligence of LLMs without compromising the natural acoustic flow. Extensive experiments show that TurnGuide not only significantly improves e2e FD-SLMs to produce semantically meaningful, coherent speech but also achieves state-of-the-art performance on various turn-taking events. Demos are available at https://dreamtheater123.github.io/TurnGuide-Demo/. Code is available at https://github.com/dreamtheater123/TurnGuide.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
