Language model agents are increasingly used to automate scientific research, yet evaluating their scientific contributions remains a challenge. A key mechanism to obtain such insights is through ablation experiments. To this end, we introduce AblationBench, a benchmark suite for evaluating agents on ablation planning tasks in empirical AI research. It includes two tasks: AuthorAblation, which helps authors propose ablation experiments based on a method section and contains 83 instances, and ReviewerAblation, which helps reviewers find missing ablations in a full paper and contains 350 instances. For both tasks, we develop LM-based judges that serve as an automatic evaluation framework. Our experiments with frontier LMs show that these tasks remain challenging, with the best-performing LM system identifying only 45% of the original ablations on average, below human-level performance. We observe an inverse performance trend between the author and reviewer tasks, which we attribute to differences in model grounding. Lastly, we analyze the limitations of current LMs on these tasks, and find that chain-of-thought prompting outperforms an agent-based approach. Our data is available on https://huggingface.co/collections/ai-coscientist/ablationbench, and our code is available on https://github.com/ai-scientist-bench/ablation-bench .
CityPlanner: A Sandbox Agent for Executable Urban Planning
School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机科学与工程学院）；MIIT Key Laboratory of Data and Decision Intelligence, Beihang University（北京航空航天大学工业和信息化部数据与决策智能重点实验室）
More Than Meets the Eye? Uncovering the Reasoning-Planning Disconnect in Training Vision-Language Driving Models
S-Lab, Nanyang Technological University, Singapore（南洋理工大学S实验室）；College of Computing and Data Science, Nanyang Technological University, Singapore（南洋理工大学计算与数据科学学院）
