MTR-DuplexBench：面向全双工语音语言模型多轮对话全面评估的综合评测
MTR-DuplexBench: Towards a Comprehensive Evaluation of Multi-Round Conversations for Full-Duplex Speech Language Models
AI总结：
本文提出MTR-DuplexBench，用于评估全双工语音语言模型在多轮对话中的表现，涵盖对话质量、指令遵循和安全性等关键方面，揭示现有模型在多轮对话中的一致性问题。
摘要：
全双工语音语言模型（FD-SLMs）能够实现实时重叠对话交互，提供比传统半双工模型更动态的用户体验。然而，现有评测基准主要关注单轮交互，忽视了多轮通信的复杂性。评估FD-SLMs在多轮设置中的挑战包括对话轮次边界模糊和模型推理中的上下文不一致。此外，现有基准仅关注对话特征，忽略了其他关键方面。为此，我们引入MTR-DuplexBench，一个新型基准，用于对FD-SLMs进行全面多轮评估。MTR-DuplexBench不仅将连续全双工对话分割为离散轮次以进行逐轮评估，还整合了包括对话特征、对话质量、指令遵循和安全性的多种评估维度。实验结果表明，当前FD-SLMs在多轮对话中难以保持一致的性能，突显了本基准的必要性和有效性。代码和数据可在：https://github.com/ZhangHe0918/MTR-DuplexBench获取。
英文摘要：
Full-Duplex Speech Language Models (FD-SLMs) enable real-time, overlapping conversational interactions, offering a more dynamic user experience compared to traditional half-duplex models. However, existing benchmarks primarily focus on evaluating single-round interactions, neglecting the complexities of multi-round communication. Evaluating FD-SLMs in multi-round settings poses significant challenges, including blurred turn boundaries in communication and context inconsistency during model inference. Also, existing benchmarks often focus solely on evaluating conversational features, neglecting other critical aspects. To address these gaps, we introduce MTR-DuplexBench, a novel benchmark designed for a comprehensive multi-round evaluation of FD-SLMs. MTR-DuplexBench not only segments continuous full-duplex dialogues into discrete turns for turn-by-turn assessment but also incorporates various evaluation aspects, including conversational features, dialogue quality, instruction following, and safety. Experimental results reveal that current FD-SLMs face difficulties in maintaining consistent performance across multiple rounds and evaluation dimensions, highlighting the necessity and effectiveness of our benchmark. Code and data are available at: https://github.com/ZhangHe0918/MTR-DuplexBench
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
