Autonomous research systems capable of generating complete scientific manuscripts have advanced rapidly, yet robust and realistic evaluation frameworks have failed to keep pace. To bridge this gap, we introduce MLReplicate, an end-to-end benchmark evaluating autonomous research systems on machine learning reproducibility. The benchmark was constructed from ICML 2025 outstanding papers reformulated into standardized input specifications and evaluated across 6 state-of-the-art research systems: AI SCIENTIST-V1, AI SCIENTIST-V2, AGENT LABORATORY, CYCLERESEARCHER, AI RESEARCHER, and TINY SCIENTIST, yielding 45 generated manuscripts, with 3 failed experiments. Outputs are assessed using a dual-protocol approach that combines automated conference-style review and structured expert human evaluation, while tracking computational cost, runtime, and the amount of required human intervention. The automated conference-style review accepted 10 out of 37 valid submissions. An additional 8 submissions were desk-rejected before review for failing to meet the minimum page threshold. In contrast to automated reviews, human reviewers consistently identified methodological flaws, hallucinated experimental results, and reproducibility failures across all systems, and 59% of accepted automated reviews contained fabricated or unsupported claims. We further find that neither token budget nor computational cost predicts output quality: the cheapest system outperforms the most resource-intensive system in human evaluation, despite a 38-fold difference in input tokens. We thus demonstrate that autonomous research workflow design matters more than the scale of compute. MLReplicate exposes a substantial gap between current autonomous research systems and genuine scientific rigor, and establishes a practical, extensible evaluation framework for systematic progress toward trustworthy AI-driven scientific discovery.
Benchmarking for Deep Uplift Modeling in Online Marketing
Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)(广东省人工智能与数字经济实验室(深圳));Tencent(腾讯);Shenzhen University(深圳大学);Renmin University of China(中国人民大学);Shenzhen Technology University(深圳技术大学)
Benchmarking of EEG Analysis Techniques for Parkinson's Disease Diagnosis: A Comparison between Traditional ML Methods and Foundation DL Methods
Department of Computer Science, Sapienza University of Rome(罗马大学计算机科学系);Fondazione Ugo Bordoni(乌戈·博尔迪尼基金会);Department of Computer Science and Mathematics, The L. H. and W. L. Collins College of Professional Studies, St. John’s University(圣约翰大学计算机科学与数学系);University of Lorraine, CNRS, CRAN Nancy(洛林大学,CNRS,Nancy CRAN)
ContextGNN goes to Elliot: Towards Benchmarking Relational Deep Learning for Static Link Prediction (aka Personalized Item Recommendation)
University of Barcelona(巴塞罗那大学);University of Southern California(南加州大学);Université Paris-Saclay(巴黎萨克雷大学);CentraleSupélec(中央理工-高等电力学院);Inria(法国国家信息与自动化研究所)
RWTH Aachen University(亚琛RWTH大学);University of Oxford(牛津大学);Mila – Quebec AI Institute(魁北克AI研究所);Technion - Israel Institute of Technology(技术学院-以色列理工学院);ETAS Research University of Stuttgart(斯图加特大学ETAS研究所);Google Research(谷歌研究);Microsoft Research(微软研究院)
A Sensing Dataset Protocol for Benchmarking and Multi-Task Wireless Sensing
School of Information and Communication Engineering, Beijing University of Posts and Telecommunications(信息与通信工程学院,北京邮电大学);College of Electronics and Information Engineering, Shenzhen University(电子与信息工程学院,深圳大学);Huawei Technology Co. Ltd.(华为技术有限公司);Department of Electronic and Electrical Engineering, University College London(电子与电气工程系,伦敦大学学院)
Benchmarking Deep Learning-Based Reconstruction Methods for Photoacoustic Computed Tomography with Clinically Relevant Synthetic Datasets
Department of Bioengineering, University of Illinois Urbana-Champaign(生物工程系,伊利诺伊大学厄巴纳-香槟分校);Department of Electrical and Computer Engineering, University of Illinois Urbana-Champaign(电气与计算机工程系,伊利诺伊大学厄巴纳-香槟分校);Department of Biomedical Engineering, The University of Texas at Austin(生物医学工程系,德克萨斯大学奥斯汀分校);Oden Institute for Computational Engineering and Sciences, The University of Texas at Austin(奥登计算工程与科学研究所,德克萨斯大学奥斯汀分校)