arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2610.22155cs.AI
收藏

Realtime-Bench:前沿模型能解决实时交互问题吗?

Realtime-Bench: Can Frontier Models Solve Real-time Interactive Problems?

University of Texas at Austin(得克萨斯大学奥斯汀分校);The Ohio State University(俄亥俄州立大学);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);NVIDIA(英伟达);Northwestern University(西北大学);Steadyworks

Jingyu Tang, Xu Yang, Zhehao Zhang, Zheng Wang, Qi Cheng, Haozhe Zhao, Manling Li, Wenxuan Li, Xide Xia, Jiaxin Pei

AI总结:

该研究推出Realtime-Bench基准,测试发现前沿模型在实时交互任务中表现不佳,时间成本限制其成功,为改进实时多模态智能体提供方向。

摘要:

实时交互要求智能体感知动态变化的环境并在时间约束下行动,但现有对实时流式模型的评估聚焦于对话质量和流式感知,未测试严格时间约束下的有效行动。我们推出Realtime-Bench,一套精心设计的动态任务,其中延迟行动直接影响成功与否。在Wall Clock(环境按实际推理延迟推进)设置下,最佳成功率仅为44%,最佳Pass^3仅为36%,表明实时交互仍具挑战性。为分离服务延迟与模型的实时流式能力,我们对比Wall Clock与Paused Clock(消除推理延迟的理想参考),并引入Token Clock(按生成token数推进模拟时间,独立于物理服务速度测试时间感知)。我们发现时间成本限制动态环境中的成功,且更高的推理努力在时间压力下效果不一,反映决策质量与及时执行间的权衡。这些发现为改进实时多模态智能体指明方向,包括具备时间感知的自适应推理、持续感知与行动。

英文摘要:

Real-time interaction requires agents to perceive evolving environments and act under temporal constraints, yet existing evaluations of real-time streaming models focus on conversational quality and streaming perception, leaving effective action under strict temporal constraints untested. We introduce Realtime-Bench, a suite of carefully curated dynamic tasks where delayed actions directly affect success. Under Wall Clock, where environments advance according to actual inference latency, the best success rate reaches only 44% and the best Pass^3 is only 36%, demonstrating that real-time interaction remains challenging. To separate serving latency from models' real-time streaming capability, we compare Wall Clock with Paused Clock, which removes inference delays as an idealized reference, and introduce Token Clock, which advances simulated time according to generated token counts, testing temporal awareness independently of physical serving speed. We find that temporal costs limit success in evolving environments and that higher reasoning effort has mixed effects under time pressure, reflecting a trade-off between decision quality and timely execution. Based on these results, we discuss future directions for improving real-time multimodal models, including adaptive reasoning with temporal awareness and continuous perception and action.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑