AI models are increasingly deployed in live clinical environments where they must perform reliably across complex, high-stakes workflows that standard training and validation datasets were never designed to capture. Evaluating these systems requires benchmarks: structured combinations of tasks, datasets, and metrics that enable reproducible, comparable measurement of what a model can do. The central challenge in healthcare AI is not performance alone, but the absence of systematic methods to measure reliability, safety, and clinical relevance under real-world conditions. Most existing benchmarks test what a model knows; too few test whether it can perform reliably and without failing across the full complexity of real clinical tasks. Current benchmarks have accumulated through ad hoc dataset construction optimized for narrow task performance: frontier models achieve near-perfect scores on medical licensing examinations, but when evaluated across real clinical tasks, performance degrades sharply, scoring 0.74--0.85 on documentation, 0.61--0.76 on clinical decision support, and only 0.53--0.63 on administrative and workflow tasks \cite{medhelm}. High benchmark scores give a false sense of deployment readiness, and the gap between performance and utility widens precisely as AI systems take on more consequential clinical roles. Without a principled framework for benchmark design, the field cannot determine whether poor clinical performance reflects model limitations or failures in how performance is being measured.
AgentRecBench: Benchmarking LLM Agent-based Personalized Recommender Systems
Tsinghua University（清华大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；University of Science and Technology of China（中国科学技术大学）；National University of Singapore（新加坡国立大学）
Grounded by Experience: Generative Healthcare Prediction Augmented with Hierarchical Agentic Retrieval
Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology（电子与计算机工程系，香港科技大学）；College of Management and Economics, Laboratory of Computation and Analytics of Complex Management Systems (CACMS), Tianjin University（管理学院与经济学学院，复杂管理系统的计算与分析实验室，天津大学）；Department of Computer Science and Engineering, The Hong Kong University of Science and Technology（计算机科学与工程系，香港科技大学）
AgenticShop: Benchmarking Agentic Product Curation for Personalized Web Shopping
Department of Artificial Intelligence Yonsei University Seoul Republic of Korea（人工智能系 首尔国立庆尚大学 首尔 大韩民国）；ParamitaAI Seoul Republic of Korea（ParamitaAI 首尔 大韩民国）；Yonsei University（首尔国立庆尚大学）；ParamitaAI
DeepPHY: Benchmarking Agentic VLMs on Physical Reasoning
1 Taobao \& Tmall Group of Alibaba, 2 Institute of Software, Chinese Academy of Science, 3 University of Chinese Academy of Sciences, 4 Renmin University of China, 5 Informatics Department, PUC-Rio
SciHorizon-eLab: An Agentic Protocol-to-Task Compiler for Scalable Benchmarking of Scientific Embodied Agents
Data Intelligence for Scientific Innovation Lab, Computer Network Information Center, Chinese Academy of Sciences（中国科学院计算机网络信息中心科学智能创新数据实验室）；Beihang University（北京航空航天大学）
BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games
University College London（伦敦大学学院）；IDEAS NCBR；University of Oxford（牛津大学）；New York University（纽约大学）；Anthropic；University of Warsaw（华沙大学）；Institute of Mathematics, Polish Academy of Sciences（波兰科学院数学研究所）
Agentic AI-based Framework for Mitigating Premature Diagnostic Handoff and Silent Hallucination in Healthcare Applications
Distributed Systems (qCLOUDS) Lab, School of Computing；Information Systems, The University of Melbourne, Australia；2Department of Computer Science；Engineering, School of Electrical；Computer Sciences (SECS), Indian Institute of Technology Bhubaneswar, India；3Department of Computer Science Banaras Hindu University, Varanasi, India
SW-$A^2$-Bench: Benchmarking Autonomous Software Agent Generation for Agentic Web
Shanghai Jiao Tong University（上海交通大学）；The University of Tokyo（东京大学）；Huazhong University of Science and Technology（华中科技大学）；Shanghai Innovation Institute（上海创新研究院）；Nankai University（南开大学）；Singapore University of Technology and Design（新加坡科技设计大学）；Queen’s University（女王大学）；Fudan University（复旦大学）；Zhejiang University（浙江大学）
LocalSearchBench: Benchmarking Agentic Search in Real-World Local Life Services
Meituan, Beijing, China（美团，北京，中国）；East China Normal University Shanghai Innovation Institute（东华大学上海创新研究院）；University of Science and Technology of China（中国科学技术大学）；Shanghai Jiaotong University（上海交通大学）；North China University of Technology, Beijing, China（华北理工大学，北京，中国）；East China Normal University Shanghai China（东华大学上海）
Picking the Right Specialist: Attentive Neural Process-based Selection of Task-Specialized Models as Tools for Agentic Healthcare Systems
Department of Engineering Science, University of Oxford, United Kingdom（牛津大学工程科学系）；Department of Computer Science, Khalifa University, Abu Dhabi, United Arab Emirates（哈利法大学计算机科学系）
HealthSLM-Bench: Benchmarking Small Language Models for Mobile and Wearable Healthcare Monitoring
School of Computing and Information Systems, University of Melbourne, Australia（墨尔本大学计算机与信息系统学院）；School of Computer Science, University of Auckland, New Zealand（奥克兰大学计算机科学学院）
StarDojo: Benchmarking Open-Ended Behaviors of Agentic Multimodal LLMs in Production-Living Simulations with Stardew Valley
Nanyang Technological University（南洋理工大学）；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；East China University of Science and Technology（华东理工大学）；Singapore Management University（新加坡管理大学）
A Closer Look at Agentic BBO: Benchmarking LLM Agents for Black-Box Optimization
State Key Laboratory of Novel Software Technology, Nanjing University（南京大学现代软件工程国家重点实验室）；School of Artificial Intelligence, Nanjing University（南京大学人工智能学院）
BackendForge: Benchmarking Agentic End-to-End Code Generation with Backend Services
Key Lab of HCST (PKU), MOE（北京大学计算机科学与技术国家重点实验室；软件学院，北京大学）；SCS, Peking University（北京通明湖信息技术应用创新中心）；Beijing Tongming Lake Information Technology Application Innovation Center (TLAIC)（复旦大学高级计算系统研究所）；Fudan University Institute of Systems for Advanced Computing（上海开放计算系统研究所）；Shanghai Institute of Systems for Open Computing
