音频多挑战：对语音对话系统在自然人类交互中的多轮评估
Audio MultiChallenge: A Multi-Turn Evaluation of Spoken Dialogue Systems on Natural Human Interaction
AI总结：
Audio MultiChallenge通过多轮交互评估语音对话系统在自然对话中的表现，揭示模型在多轮对话、语音修复和长上下文处理上的挑战。
摘要：
端到端（E2E）语音对话系统正越来越多地取代级联管道，用于基于语音的人机交互，直接处理原始音频而不经过中间转录。现有的基准测试主要评估这些模型在合成语音和单轮任务上的表现，而对现实中的多轮对话能力研究不足。我们引入了Audio MultiChallenge，一个开源基准测试，用于评估端到端语音对话系统在自然多轮交互模式下的表现。基于文本-based的MultiChallenge框架，该框架评估推理记忆、指令保留和自我一致性，我们引入了一个新的轴Voice Editing，测试模型在中间句子语音修复和回溯时的鲁棒性。我们进一步将每个轴扩展到音频模式，例如引入Audio-Cue挑战来评估推理记忆，要求在语义内容之外回忆环境声音和副语言信号。我们通过混合的音频原生代理和人工在环管道，从47名说话者中精选了452次对话，通过1712个实例特定的评分标准，使模型在大规模下暴露故障，同时保留非剧本人类语音中的自然不流畅性。我们对专有和开源模型的评估显示，即使前沿模型在我们的基准测试中也表现不佳，我们的最高性能模型Gemini 3 Pro Preview（Thinking）仅达到54.65%的通过率。错误分析表明，模型在我们新轴上失败最频繁，且自我一致性随着音频上下文的延长而下降。这些失败反映了在自然语音对话中跟踪编辑、音频提示和长距离上下文的难度。Audio MultiChallenge提供了一个可重复测试的平台，以量化这些挑战并推动音频原生多轮交互能力的改进。
英文摘要：
End-to-end (E2E) spoken dialogue systems are increasingly replacing cascaded pipelines for voice-based human-AI interaction, processing raw audio directly without intermediate transcription. Existing benchmarks primarily evaluate these models on synthetic speech and single-turn tasks, leaving realistic multi-turn conversational ability underexplored. We introduce Audio MultiChallenge, an open-source benchmark to evaluate E2E spoken dialogue systems under natural multi-turn interaction patterns. Building on the text-based MultiChallenge framework, which evaluates Inference Memory, Instruction Retention, and Self Coherence, we introduce a new axis Voice Editing that tests robustness to mid-utterance speech repairs and backtracking. We further augment each axis to the audio modality, such as introducing Audio-Cue challenges for Inference Memory that require recalling ambient sounds and paralinguistic signals beyond semantic content. We curate 452 conversations from 47 speakers with 1,712 instance-specific rubrics through a hybrid audio-native agentic and human-in-the-loop pipeline that exposes model failures at scale while preserving natural disfluencies found in unscripted human speech. Our evaluation of proprietary and open-source models reveals that even frontier models struggle on our benchmark, with Gemini 3 Pro Preview (Thinking), our highest-performing model achieving a 54.65% pass rate. Error analysis shows that models fail most often on our new axes and that Self Coherence degrades with longer audio context. These failures reflect difficulty of tracking edits, audio cues, and long-range context in natural spoken dialogue. Audio MultiChallenge provides a reproducible testbed to quantify them and drive improvements in audio-native multi-turn interaction capability.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
