arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.13686cs.SDcs.AI
收藏

$τ$-Voice:在真实领域上评估全双工语音代理的基准测试

$τ$-Voice: Benchmarking Full-Duplex Voice Agents on Real-World Domains

Soham Ray, Keshav Dhandhania, Victor Barres, Karthik Narasimhan

AI总结:

本文提出$τ$-voice基准测试,评估语音代理在真实复杂任务中的表现,发现语音代理在清洁条件和嘈杂环境下任务完成率仅为31-51%和26-38%,远低于文本能力,表明代理行为是失败的主要原因。

摘要:

全双工语音代理--能够同时听和说的系统--正从研究走向生产。然而,现有评估仅关注对话动态和任务完成。我们引入$τ$-voice,一个评估语音代理在真实任务中表现的基准:代理必须在复杂多轮对话中导航,遵守领域政策,并与环境交互。框架将$τ^2$-bench扩展为新的语音代理基准,结合可验证的复杂任务完成、全双工交互和真实音频,实现语音与文本性能的直接比较。可控且真实的语音用户模拟器提供多样化的口音、真实音频环境和丰富的轮流动态;通过将模拟与实际时间解耦,模拟器可利用最强大的LLM而不受实时限制。我们评估了278个任务的任务完成(pass@1)和语音交互质量:虽然GPT-5(推理)达到85%,语音代理在清洁条件下仅达到31-51%,在嘈杂和多样口音条件下仅26-38%,保留了文本能力的30-45%;定性分析证实79-90%的失败源于代理行为,表明观察到的失败主要反映代理在本评估设置下的行为。$τ$-voice提供了一个可重复的测试平台,用于衡量向自然、对话和可靠的语音代理进步的进展。

英文摘要:

Full-duplex voice agents--systems that listen and speak simultaneously--are rapidly moving from research to production. However, existing evaluations address conversational dynamics and task completion in isolation. We introduce $τ$-voice, a benchmark for evaluating voice agents on grounded tasks with real-world complexity: agents must navigate complex multi-turn conversations, adhere to domain policies, and interact with the environment. The framework extends $τ^2$-bench into a novel voice agent benchmark combining verifiable completion of complex grounded tasks, full-duplex interaction, and realistic audio--enabling direct comparison between voice and text performance. A controllable and realistic voice user simulator provides diverse accents, realistic audio environments, and rich turn-taking dynamics; by decoupling simulation from wall-clock time, the user simulator can use the most capable LLM without real-time constraints. We evaluate task completion (pass@1) and voice interaction quality across 278 tasks: while GPT-5 (reasoning) achieves 85%, voice agents reach only 31--51% under clean conditions and 26--38% under realistic conditions with noise and diverse accents--retaining only 30--45% of text capability; qualitative analysis confirms 79--90% of failures stem from agent behavior, suggesting that observed failures primarily reflect agent behavior under our evaluation setup. $τ$-voice provides a reproducible testbed for measuring progress toward voice agents that are natural, conversational, and reliable.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑