故事并非科学：基于执行的机制可解释性研究评估
The Story is Not the Science: Execution-Grounded Evaluation of Mechanistic Interpretability Research
AI总结：
本文提出基于执行的评估框架，通过代码和数据验证机制可解释性研究的严谨性，发现大量方法学问题，展示AI在研究评估中的潜力。
摘要：
各学科的可重复性危机凸显了以论文为中心的审查系统在评估研究严谨性和可重复性方面的局限性。能够自主设计和生成大量研究输出的AI代理加剧了这些挑战。在本工作中，我们通过将动态发展的AI代理翻转为研究评估者，来解决可扩展性和严谨性日益增长的挑战。我们提出了首个基于执行的评估框架，通过检查代码和数据与论文一同，验证研究超越叙述性审查。我们以机制可解释性研究为测试平台，构建标准化研究输出，并开发MechEvalAgent，一个自动评估框架，评估实验过程的连贯性、结果的可重复性和发现的可推广性。我们显示，我们的框架与人类法官达成超过80%的共识，识别出显著的方法学问题，并揭示了51个被人类评审员遗漏的问题。我们的工作展示了AI代理在研究评估中的潜力，并为严谨的科学实践铺平了道路。
英文摘要：
Reproducibility crises across sciences highlight the limitations of the paper-centric review system in assessing the rigor and reproducibility of research. AI agents that autonomously design and generate large volumes of research outputs exacerbate these challenges. In this work, we address the growing challenges of scalability and rigor by flipping the dynamic and developing AI agents as research evaluators. We propose the first execution-grounded evaluation framework that verifies research beyond narrative review by examining code and data alongside the paper. We use mechanistic interpretability research as a testbed, build standardized research output, and develop MechEvalAgent, an automated evaluation framework that assesses the coherence of the experimental process, the reproducibility of results, and the generalizability of findings. We show that our framework achieves above 80% agreement with human judges, identifies substantial methodological problems, and surfaces 51 additional issues that human reviewers miss. Our work demonstrates the potential of AI agents to transform research evaluation and pave the way for rigorous scientific practices.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
