arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2606.05080cs.AIcs.LG
收藏

AutoLab:前沿模型能否解决长周期自动研究与工程任务?

AutoLab: Can Frontier Models Solve Long-Horizon Auto Research and Engineering Tasks?

MIT(麻省理工学院);Stanford University(斯坦福大学);University of California, Berkeley(加州大学伯克利分校);University of California, Los Angeles(加州大学洛杉矶分校);University of California, San Diego(加州大学圣地亚哥分校);University of Washington(华盛顿大学);University of Toronto(多伦多大学);University of Michigan(密歇根大学);National University of Singapore(新加坡国立大学);University of Tokyo(东京大学)
Zhangchen Xu, Junda Chen, Yue Huang, Dongfu Jiang, Jiefeng Chen, Hang Hua, Zijian Wu, Zheyuan Liu, Zexue He, Lichi Li, Shizhe Diao, Jiaxin Pei, Jinsung Yoon, Ha… 展开作者

Zhangchen Xu, Junda Chen, Yue Huang, Dongfu Jiang, Jiefeng Chen, Hang Hua, Zijian Wu, Zheyuan Liu, Zexue He, Lichi Li, Shizhe Diao, Jiaxin Pei, Jinsung Yoon, Hao Zhang, Mengdi Wang, Radha Poovendran, Misha Sra, Alex Pentland, Zichen Chen

AI总结:

本文提出AutoLab基准,通过36个专家策划的长周期闭环优化任务评估前沿模型,发现持续迭代和利用经验反馈比初始尝试质量更重要。

摘要:

科学和工程进步本质上是一个长周期迭代过程:提出更改、运行实验、测量结果并不断改进工件。然而,现有的前沿模型基准主要评估单轮响应或短周期智能体轨迹,未能捕捉在长时间跨度内持续迭代改进的挑战。为了解决这一差距,我们引入了AutoLab,一个用于超长周期闭环优化的新基准。AutoLab包含36个现实且由专家策划的任务,涵盖四个不同领域:系统优化、谜题与挑战、模型开发和CUDA内核优化。每个任务从一个正确但故意次优的基线开始,并挑战智能体在严格的挂钟预算内改进它。评估17个最先进模型的结果表明,成功的主要预测因素不是智能体初始尝试的质量,而是其持续进行基准测试、编辑和整合经验反馈的毅力。虽然claude-opus-4.6表现出强大的长周期优化能力,但大多数前沿模型,包括几个专有模型,要么过早终止,要么在预算内进展甚微。这些结果强调了时间意识和持续迭代在自主智能体中的重要性。我们开源了完整的基准、评估框架和任务工件,以加速研究真正有能力的长周期智能体。

英文摘要:

Scientific and engineering progress is fundamentally a long-horizon iterative process: proposing changes, running experiments, measuring outcomes, and continuously refining artifacts. Yet existing benchmarks for frontier models primarily evaluate either single-turn responses or short-horizon agent trajectories, failing to capture the challenges of sustained iterative improvement over extended time horizons. To address this gap, we introduce AutoLab, a new benchmark for ultra long-horizon closed-loop optimization. AutoLab consists of 36 realistic, expert-curated tasks spanning four diverse domains: system optimization, puzzle & challenge, model development, and CUDA kernel optimization. Each task begins with a correct but deliberately suboptimal baseline and challenges agents to improve it within a strict wall-clock budget. Evaluating 17 state-of-the-art models reveals the dominant predictor of success is not the quality of an agent's initial attempt, but its persistence in repeatedly benchmarking, editing, and incorporating empirical feedback. While claude-opus-4.6 exhibits strong long-horizon optimization capabilities, most frontier models, including several proprietary ones, either terminate prematurely or exhaust their budgets with minimal progress. These results underscore the importance of time awareness and persistent iteration in autonomous agents. We open-source the full benchmark, evaluation harness, and task artifacts, to accelerate research toward truly capable long-horizon agents.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑