AutoLab: Can Frontier Models Solve Long-Horizon Auto Research and Engineering Tasks?
MIT(麻省理工学院);Stanford University(斯坦福大学);University of California, Berkeley(加州大学伯克利分校);University of California, Los Angeles(加州大学洛杉矶分校);University of California, San Diego(加州大学圣地亚哥分校);University of Washington(华盛顿大学);University of Toronto(多伦多大学);University of Michigan(密歇根大学);National University of Singapore(新加坡国立大学);University of Tokyo(东京大学)
Scientific and engineering progress is fundamentally a long-horizon iterative process: proposing changes, running experiments, measuring outcomes, and continuously refining artifacts. Yet existing benchmarks for frontier models primarily evaluate either single-turn responses or short-horizon agent trajectories, failing to capture the challenges of sustained iterative improvement over extended time horizons. To address this gap, we introduce AutoLab, a new benchmark for ultra long-horizon closed-loop optimization. AutoLab consists of 36 realistic, expert-curated tasks spanning four diverse domains: system optimization, puzzle & challenge, model development, and CUDA kernel optimization. Each task begins with a correct but deliberately suboptimal baseline and challenges agents to improve it within a strict wall-clock budget. Evaluating 17 state-of-the-art models reveals the dominant predictor of success is not the quality of an agent's initial attempt, but its persistence in repeatedly benchmarking, editing, and incorporating empirical feedback. While claude-opus-4.6 exhibits strong long-horizon optimization capabilities, most frontier models, including several proprietary ones, either terminate prematurely or exhaust their budgets with minimal progress. These results underscore the importance of time awareness and persistent iteration in autonomous agents. We open-source the full benchmark, evaluation harness, and task artifacts, to accelerate research toward truly capable long-horizon agents.
Frontier Coding Agents Can Now Implement an AlphaZero Self-Play Machine Learning Pipeline For Connect Four That Performs Comparably to an External Solver
Department of Computer Science(计算机科学系);University of Chicago(芝加哥大学);Independent Researcher(独立研究员)
本文提出一个基准测试,评估前沿编码代理自主实现AlphaZero式机器学习流水线的能力,通过简洁任务描述而非完整前序工作参考,评估连接四游戏AI性能,发现Claude Opus 4.7在七次试验中胜过Pons求解器,其他代理表现欠佳。
Open-World Evaluations for Measuring Frontier AI Capabilities
Princeton University(普林斯顿大学);Cornflower Labs(Cornflower实验室);Meridian Labs(Meridian实验室);Stanford University(斯坦福大学);UK AI Security Institute(英国人工智能安全研究所);Johns Hopkins University(约翰霍普金斯大学);Adaption Labs(Adaption实验室);Australian National University(澳大利亚国立大学);Golden Gate Institute for AI(金门人工智能研究所);UW Madison(威斯康星大学麦迪逊分校);Microsoft Research(微软研究院);AI Digest(AI摘要);Georgetown University (CSET)(乔治城大学(CSET))
FrontierCS: Evolving Challenges for Evolving Intelligence
UC Berkeley(加州大学伯克利分校);Princeton University(普林斯顿大学);UCSD(加州大学圣迭戈分校);X-camp Academy(X-camp学院);Georgia Tech(佐治亚理工学院);Stanford University(斯坦福大学);University of Washington(华盛顿大学);Nanyang Technological University(南洋理工大学);University of Toronto(多伦多大学);UIUC(伊利诺伊大学香槟分校);University of Michigan(密歇根大学);New York University(纽约大学);MIT(麻省理工学院)
AI Mathematician: Towards Fully Automated Frontier Mathematical Research
Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University, Beijing, China(计算机科学与技术系,人工智能研究院,清华大学,北京,中国);Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China(人工智能产业研究院(AIR),清华大学,北京,中国);Department of Mathematical Sciences, Tsinghua University(数学科学系,清华大学);Qiuzhen College, Tsinghua University(齐臻学院,清华大学)
Clear, Compelling Arguments: Rethinking the Foundations of Frontier AI Safety Cases
UKRI Centre for Doctoral Training in Safe AI Systems (SAINTS)(英国研究与创新机构安全人工智能系统博士培训中心);Institute for Safe Autonomy(安全自主研究所);The York Law School(约克法律学院)
ForesightSafety Bench: A Frontier Risk Evaluation and Governance Framework towards Safe AI
Beijing Institute of AI Safety and Governance(北京人工智能安全与治理研究院);Beijing Key Laboratory of Safe AI and Superalignment(北京安全人工智能与超对齐重点实验室);BrainCog Lab, Institute of Automation, Chinese Academy of Sciences(脑认知实验室,中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);Long-term AI(长期人工智能)