arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.15112cs.AI
收藏

ResearchGym: 在真实世界人工智能研究中评估语言模型代理

ResearchGym: Evaluating Language Model Agents on Real-World AI Research

Aniketh Garikaparthi, Manasi Patwardhan, Arman Cohan

AI总结:

ResearchGym通过重新利用学术论文中的任务环境,评估语言模型代理在真实世界AI研究中的能力,发现其在复杂任务中存在可靠性不足的问题,但偶尔能实现前沿性能。

摘要:

我们介绍了ResearchGym,一个用于评估AI代理进行端到端研究的基准和执行环境。为此,我们重新利用了来自ICML、ICLR和ACL会议的五篇口头报告和亮点论文。从每篇论文的存储库中,我们保留了数据集、评估工具和基线实现,但不包含论文中提出的方法。这导致了五个容器化任务环境,总共包含39个子任务。在每个环境中,代理必须提出新的假设、运行实验,并尝试在论文的指标上超越强大的人类基线。在对一个由GPT-5驱动的代理进行受控评估时,我们观察到一种明显的能力-可靠性差距。该代理在15次评估中的1次(6.7%)中提升了11.5%,平均只完成了26.5%的子任务。我们识别出反复出现的长周期失败模式,包括急躁、时间与资源管理不佳、对弱假设过于自信、难以协调并行实验以及上下文长度的硬限制。然而,在一次运行中,该代理超越了ICML 2025亮点任务的解决方案,表明前沿代理偶尔可以达到最先进的性能,但这种表现不可靠。我们还评估了专有代理框架,包括Claude Code(Opus-4.5)和Codex(GPT-5.2),它们显示出相似的差距。ResearchGym提供了系统评估和分析自主代理在闭环研究中基础设施的平台。

英文摘要:

We introduce ResearchGym, a benchmark and execution environment for evaluating AI agents on end-to-end research. To instantiate this, we repurpose five oral and spotlight papers from ICML, ICLR, and ACL. From each paper's repository, we preserve the datasets, evaluation harness, and baseline implementations but withhold the paper's proposed method. This results in five containerized task environments comprising 39 sub-tasks in total. Within each environment, agents must propose novel hypotheses, run experiments, and attempt to surpass strong human baselines on the paper's metrics. In a controlled evaluation of an agent powered by GPT-5, we observe a sharp capability--reliability gap. The agent improves over the provided baselines from the repository in just 1 of 15 evaluations (6.7%) by 11.5%, and completes only 26.5% of sub-tasks on average. We identify recurring long-horizon failure modes, including impatience, poor time and resource management, overconfidence in weak hypotheses, difficulty coordinating parallel experiments, and hard limits from context length. Yet in a single run, the agent surpasses the solution of an ICML 2025 Spotlight task, indicating that frontier agents can occasionally reach state-of-the-art performance, but do so unreliably. We additionally evaluate proprietary agent scaffolds including Claude Code (Opus-4.5) and Codex (GPT-5.2) which display a similar gap. ResearchGym provides infrastructure for systematic evaluation and analysis of autonomous agents on closed-loop research.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑