全双工语音模型中的多面交互对齐
Multi-Faceted Interactivity Alignment in Full-Duplex Speech Models
AI总结：
针对全双工对话模型交互性问题，提出基于强化学习的后训练对齐方法，从暂停处理、话轮转换、回馈和用户打断四个维度优化，并加入LLM奖励防止语义退化，在Moshi和PersonaPlex上取得一致改进。
摘要：
全双工口语对话模型可以同时听和说，使其成为自然对话的有前途的架构。然而，当前模型仅通过令牌级似然最大化的监督学习进行训练，这并未直接优化交互级行为，导致交互性问题，如过度沉默和不合时宜的话轮转换。最近的工作应用强化学习（RL）来改善交互性，但现有方法仅在其奖励中处理有限的一组交互行为。在这项工作中，我们提出了一种后训练对齐方法，通过RL全面改善全双工口语对话模型的交互性。我们解决了交互性的四个典型轴：暂停处理、话轮转换、回馈和用户打断。对于每个轴，我们从人类对话语料库中提取短音频片段，并使用特定于轴的奖励函数优化模型。一个额外的基于LLM的响应质量奖励防止语义退化。我们将我们的方法应用于两个开源模型Moshi和PersonaPlex，在预录音频的离线评估和实时多轮对话评估中均显示出交互性的一致改进。
英文摘要：
Full-duplex spoken dialogue models can listen and speak simultaneously, making them a promising architecture for natural conversation. However, current models are trained solely with supervised learning through token-level likelihood maximization, which does not directly optimize interaction-level behaviors, causing interactivity issues such as excessive silence and ill-timed turn-taking. Recent work has applied reinforcement learning (RL) to improve interactivity, but existing methods address only a limited set of interactive behaviors in their rewards. In this work, we propose a post-training alignment method that comprehensively improves the interactivity of full-duplex spoken dialogue models through RL. We address the four canonical axes of interactivity: pause handling, turn-taking, backchanneling, and user interruption. For each axis, we extract short audio segments from human conversation corpora and optimize the model with axis-specific reward functions. An extra LLM-based reward for response quality prevents semantic degradation. We apply our method to two open-source models, Moshi and PersonaPlex, demonstrating consistent improvements in interactivity on both offline evaluation with pre-recorded audio and real-time multi-turn dialogue evaluation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
