Realtime-Bench: Can Frontier Models Solve Real-time Interactive Problems?
University of Texas at Austin(得克萨斯大学奥斯汀分校);The Ohio State University(俄亥俄州立大学);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);NVIDIA(英伟达);Northwestern University(西北大学);Steadyworks
Real-time interaction requires agents to perceive evolving environments and act under temporal constraints, yet existing evaluations of real-time streaming models focus on conversational quality and streaming perception, leaving effective action under strict temporal constraints untested. We introduce Realtime-Bench, a suite of carefully curated dynamic tasks where delayed actions directly affect success. Under Wall Clock, where environments advance according to actual inference latency, the best success rate reaches only 44% and the best Pass^3 is only 36%, demonstrating that real-time interaction remains challenging. To separate serving latency from models' real-time streaming capability, we compare Wall Clock with Paused Clock, which removes inference delays as an idealized reference, and introduce Token Clock, which advances simulated time according to generated token counts, testing temporal awareness independently of physical serving speed. We find that temporal costs limit success in evolving environments and that higher reasoning effort has mixed effects under time pressure, reflecting a trade-off between decision quality and timely execution. Based on these results, we discuss future directions for improving real-time multimodal models, including adaptive reasoning with temporal awareness and continuous perception and action.
FrontierMat: Benchmarking Language Models at the Frontier of Materials R&D
Nanyang Technological University(南洋理工大学);Peking University(北京大学);Shanghai Jiao Tong University(上海交通大学);Xi’an Jiaotong University(西安交通大学);Wuhan University of Technology(武汉理工大学);Beihang University(北京航空航天大学);Eighth Research Institute of Nuclear Industry, CNNC(中国核工业集团公司第八研究所);Harbin Institute of Technology(哈尔滨工业大学);Tsinghua University(清华大学);University of Science and Technology of China(中国科学技术大学);Commercial Aircraft Corporation of China (COMAC)(中国商飞公司);Tongji University(同济大学);Northwestern Polytechnical University(西北工业大学);Wanhua Chemical Group Co., Ltd.(万华化学集团股份有限公司)
CIAware-Bench: Benchmarking Control Intervention Awareness Across Frontier LLMs
MATS;Mila – Quebec AI Institute(Mila – 魁北克人工智能研究所);Université de Montréal(蒙特利尔大学);Astra Fellowship;ELLIS Institute Tübingen, MPI for Intelligent Systems & Tübingen AI Center;LawZero;Google DeepMind(谷歌DeepMind)
ForesightSafety Bench: A Frontier Risk Evaluation and Governance Framework towards Safe AI
Beijing Institute of AI Safety and Governance(北京人工智能安全与治理研究院);Beijing Key Laboratory of Safe AI and Superalignment(北京安全人工智能与超对齐重点实验室);BrainCog Lab, Institute of Automation, Chinese Academy of Sciences(脑认知实验室,中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);Long-term AI(长期人工智能)
PRL-Bench: A Comprehensive Benchmark Evaluating LLMs' Capabilities in Frontier Physics Research
School of Artificial Intelligence(人工智能学院);Zhiyuan College(智远学院);School of Physics and Astronomy(物理天文学院);Tsung-Dao Lee Institute(李政道研究所);State Key Laboratory of Dark Matter Physics(暗物质物理国家重点实验室);Shanghai Innovation Institute(上海创新研究院);Institute of Theoretical Physics(理论物理研究所);Chinese Academy of Sciences(中国科学院);Zhejiang Lab(浙江实验室);SciLand;DP Technology(DP技术)
Systematic Capability Benchmarking of Frontier Large Language Models for Offensive Cyber Tasks
Department of Cybersecurity(网络安全系);Department of Information Sciences and Technology(信息科学与技术系);College of Emergency Preparedness, Homeland Security, and Cybersecurity(应急准备、国土安全与网络安全学院);University at Albany, SUNY(阿尔巴尼大学)
AutoLab: Can Frontier Models Solve Long-Horizon Auto Research and Engineering Tasks?
MIT(麻省理工学院);Stanford University(斯坦福大学);University of California, Berkeley(加州大学伯克利分校);University of California, Los Angeles(加州大学洛杉矶分校);University of California, San Diego(加州大学圣地亚哥分校);University of Washington(华盛顿大学);University of Toronto(多伦多大学);University of Michigan(密歇根大学);National University of Singapore(新加坡国立大学);University of Tokyo(东京大学)
Budget-Aware LLM Discovery via Cost-Calibrated Frontier Utility
City University of Hong Kong(香港城市大学);Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)(穆罕默德·本·扎耶德人工智能大学);International Digital Economy Academy (IDEA)(国际数字经济学院)