arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2605.16616cs.LG
收藏

MLReplicate:用于机器学习可重复性自主研究系统的基准测试

MLReplicate: Benchmarking Autonomous Research Systems for Machine Learning Reproducibility

TIB—Leibniz Information Centre for Science and Technology(蒂宾根-莱比锡信息科学与技术研究中心);L3S Research Center, Leibniz University Hannover(莱比锡大学汉诺威分校L3S研究中心)

Sasi Kiran Gaddipati, Diyana Muhammed, Farhana Keya, Gollam Rabby, Sören Auer

AI总结:

本文提出MLReplicate基准测试,评估自主研究系统在机器学习可重复性方面的性能,发现计算规模并非决定性因素,且系统间存在显著差距。

摘要:

能够生成完整科学论文的自主研究系统已取得显著进展,但稳健且真实的评估框架仍未跟上。为弥合这一差距,我们引入MLReplicate,一个端到端的基准测试,评估自主研究系统在机器学习可重复性方面的表现。该基准测试基于ICML 2025中表现突出的论文,重新制定为标准化的输入规范,并在六个最先进的研究系统上进行评估:AI SCIENTIST-V1、AI SCIENTIST-V2、AGENT LABORATORY、CYCLERESEARCHER、AI RESEARCHER和TINY SCIENTIST,生成了45篇论文,其中3篇实验失败。输出通过双协议方法评估,结合自动化会议式评审和结构化专家人工评估,同时跟踪计算成本、运行时间和所需的人类干预量。自动化会议式评审接受了37份有效提交中的10份。另外8份提交在评审前被桌面拒绝,因未达到最低页面阈值。与自动化评审相比,人工评审员一致发现所有系统中存在方法学缺陷、幻觉实验结果和可重复性失败,并且59%的接受自动化评审包含伪造或未经证实的声明。我们进一步发现,token预算和计算成本并不能预测输出质量:最便宜的系统在人工评估中优于最资源密集的系统,尽管输入token数量相差38倍。因此,我们证明自主研究工作流设计比计算规模更重要。MLReplicate揭示了当前自主研究系统与真实科学严谨性之间的显著差距,并建立了一个实用且可扩展的评估框架,以系统性推进可信的人工智能驱动科学发现。

英文摘要:

Autonomous research systems capable of generating complete scientific manuscripts have advanced rapidly, yet robust and realistic evaluation frameworks have failed to keep pace. To bridge this gap, we introduce MLReplicate, an end-to-end benchmark evaluating autonomous research systems on machine learning reproducibility. The benchmark was constructed from ICML 2025 outstanding papers reformulated into standardized input specifications and evaluated across 6 state-of-the-art research systems: AI SCIENTIST-V1, AI SCIENTIST-V2, AGENT LABORATORY, CYCLERESEARCHER, AI RESEARCHER, and TINY SCIENTIST, yielding 45 generated manuscripts, with 3 failed experiments. Outputs are assessed using a dual-protocol approach that combines automated conference-style review and structured expert human evaluation, while tracking computational cost, runtime, and the amount of required human intervention. The automated conference-style review accepted 10 out of 37 valid submissions. An additional 8 submissions were desk-rejected before review for failing to meet the minimum page threshold. In contrast to automated reviews, human reviewers consistently identified methodological flaws, hallucinated experimental results, and reproducibility failures across all systems, and 59% of accepted automated reviews contained fabricated or unsupported claims. We further find that neither token budget nor computational cost predicts output quality: the cheapest system outperforms the most resource-intensive system in human evaluation, despite a 38-fold difference in input tokens. We thus demonstrate that autonomous research workflow design matters more than the scale of compute. MLReplicate exposes a substantial gap between current autonomous research systems and genuine scientific rigor, and establishes a practical, extensible evaluation framework for systematic progress toward trustworthy AI-driven scientific discovery.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑