AI agents are an exciting new research direction, and agent development is driven by benchmarks. Our analysis of current agent benchmarks and evaluation practices reveals several shortcomings that hinder their usefulness in real-world applications. First, there is a narrow focus on accuracy without attention to other metrics. As a result, SOTA agents are needlessly complex and costly, and the community has reached mistaken conclusions about the sources of accuracy gains. Our focus on cost in addition to accuracy motivates the new goal of jointly optimizing the two metrics. We design and implement one such optimization, showing its potential to greatly reduce cost while maintaining accuracy. Second, the benchmarking needs of model and downstream developers have been conflated, making it hard to identify which agent would be best suited for a particular application. Third, many agent benchmarks have inadequate holdout sets, and sometimes none at all. This has led to agents that are fragile because they take shortcuts and overfit to the benchmark in various ways. We prescribe a principled framework for avoiding overfitting. Finally, there is a lack of standardization in evaluation practices, leading to a pervasive lack of reproducibility. We hope that the steps we introduce for addressing these shortcomings will spur the development of agents that are useful in the real world and not just accurate on benchmarks.
Occam's razor is insufficient to infer the preferences of irrational agents
Future of Humanity Institute(人类未来研究所);University of Oxford(牛津大学);Vector Institute(向量研究所);University of Toronto(多伦多大学);Machine Intelligence Research Institute(机器智能研究所)
Collaborative Agent Reasoning Engineering (CARE): A Three-Party Design Methodology for Systematically Engineering AI Agents with Subject Matter Experts, Developers, and Helper Agents
Agentic AI Optimisation (AAIO): what it is, how it works, why it matters, and how to deal with it
Digital Ethics Center, Yale University(耶鲁大学数字伦理中心);Department of Legal Studies, University of Bologna(博洛尼亚大学法律研究系);Department of Computer Science(计算机科学系)
Joint Intrinsic Motivation for Coordinated Exploration in Multi-Agent Deep Reinforcement Learning
ECE Paris \& Sorbonne Universit\' e , CNRS,\ ISIR F-75005 Paris France;Sorbonne Universit\' e , CNRS, ISIR F-75005 Paris France;ECE Paris \& Sorbonne Universit\' e , CNRS,\ ISIR;Sorbonne Universit\' e , CNRS, ISIR
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Shanghai AI Laboratory(上海人工智能实验室);University of Science and Technology of China(中国科学技术大学);Institute of Computing Technology, CAS(中国科学院计算技术研究所)
BRAIN: Bayesian Reasoning via Active Inference for Agentic and Embodied Intelligence in Mobile Networks
School of Electrical Engineering and Computer Science, TU Berlin(技术大学柏林电气工程与计算机科学学院);Optical Zeitgeist Laboratory, INRS(光感知实验室,INRS);Federal Ministry of Research, Technology and Space (BMFTR, Germany)(德国联邦研究、科技与航天部)
Question's Gambit: The First Move Matters in Agentic Deep Search
University of Toronto(多伦多大学);Mila – Quebec AI Institute(米拉-魁北克人工智能研究所);University of Waterloo(滑铁卢大学);University of California, Berkeley(加州大学伯克利分校);McGill University(麦吉尔大学)
Does The Way You Plan Matter? An Empirical Study of Planning Representations for LLM Web Agents
Concordia University(康科德大学);Mila - Quebec AI Institute(魁北克人工智能研究所);University of Copenhagen(哥本哈根大学);Universite Claude Bernard Lyon(克莱尔蒙特-伯恩大学);McGill University(麦吉尔大学)
Request Order Matters: Cache-History Sensitivity in Selective KV-Cache Reuse for Rolling Agents
Carnegie Mellon University(卡内基梅隆大学);Capital One(第一资本);University of Chicago(芝加哥大学);University of California, Berkeley(加州大学伯克利分校);University of Southern California(南加州大学);University of Washington(华盛顿大学)