从用户交互中对齐口语对话模型
Aligning Spoken Dialogue Models from User Interactions
AI总结：
本文提出一种新颖的偏好对齐框架，通过构建超15万个AI标注的语音偏好对数据集并利用离线对齐方法微调全双工语音到语音模型，有效提升了实时口语对话的事实性、安全性与上下文对齐能力。
摘要：
我们提出了一种新颖的偏好对齐框架，旨在通过用户交互改善口语对话模型在实时对话中的表现。当前的偏好学习方法主要侧重于基于文本的语言模型，无法直接适应实时语音交互的复杂性，后者具有更丰富的动态特性（如打断、插话）且说话者轮次之间没有明确分割。我们利用原始多轮语音对话，构建了一个包含超过150,000个偏好对的大规模数据集，并通过AI反馈进行标注，以涵盖对语言内容和时间上下文变化的偏好。我们利用离线对齐方法微调了一个全双工自回归语音到语音模型。大量实验表明，对通用对话的反馈能持续有效地改进口语对话模型，使其产生更符合事实、更安全且上下文更对齐的交互。我们部署了微调后的模型，并进行了全面的人工评估，以评估单轮对话之外的影响。我们的发现揭示了在各种动态特性之间实现良好校准平衡的重要性，这对于自然的实时语音对话系统至关重要。
英文摘要：
We propose a novel preference alignment framework for improving spoken dialogue models on real-time conversations from user interactions. Current preference learning methods primarily focus on text-based language models, and are not directly suited to the complexities of real-time speech interactions, with richer dynamics (e.g. interruption, interjection) and no explicit segmentation between speaker turns.We create a large-scale dataset of more than 150,000 preference pairs from raw multi-turn speech conversations, annotated with AI feedback, to cover preferences over both linguistic content and temporal context variations. We leverage offline alignment methods to finetune a full-duplex autoregressive speech-to-speech model. Extensive experiments demonstrate that feedback on generic conversations can be consistently effective in improving spoken dialogue models to produce more factual, safer and more contextually aligned interactions. We deploy the finetuned model and conduct holistic human evaluations to assess the impact beyond single-turn conversations. Our findings shed light on the importance of a well-calibrated balance among various dynamics, crucial for natural real-time speech dialogue systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
