NLPCC 2026 共享任务11概览：基于智能体的科学论文实验复现

中文摘要 AI 辅助

可复现性对科学进步至关重要，然而科学出版物数量与复杂性的日益增长，使得详尽的人工验证愈发不切实际。尽管大语言模型（LLM）智能体的最新进展使得自动化实验复现成为可能，但现有评估大多聚焦于最终代码库，且通常仅限于机器学习（ML）领域。我们提出AgentActionBench，一个面向过程的基准，用于评估ML与AI4Science领域中基于智能体的实验复现。我们的框架采用基于MCP的动作记录器（Action Recorder）来捕捉智能体在整个复现过程中的行为，并使用论文特定的评分细则（rubrics）对生成的轨迹进行评估。AgentActionBench包含150篇论文，其中120篇为ML论文，30篇为AI4Science论文。一个覆盖基准10%的人工标注子集提供了验证数据，而模型辅助增强将完整基准扩展至超过10,000个评分细则条目。实验结果表明，现有系统仍存在局限，执行是主要瓶颈。同时，模型生成与人工标注评分细则之间的强Pearson和Spearman相关性验证了我们可扩展评分细则生成方法的可靠性。

英文摘要

Reproducibility is essential to scientific progress, yet the growing volume and complexity of scientific publications make exhaustive manual verification increasingly impractical. Although recent advances in large language model (LLM) agents enable automated experiment reproduction, existing evaluations largely focus on final repositories and are typically limited to machine learning (ML). We introduce AgentActionBench, a process-oriented benchmark for evaluating agent-based experiment reproduction across ML and AI4Science domains. Our framework uses an MCP-based Action Recorder to capture agents' behaviour throughout the reproduction process and evaluates the resulting traces with paper-specific rubrics. AgentActionBench contains 150 papers, including 120 ML papers and 30 AI4Science papers. A human-annotated subset covering 10% of the benchmark provides validation data, while model-assisted augmentation expands the full benchmark to more than 10,000 rubric items. Experimental results show that current systems remain limited, with execution as the primary bottleneck. Meanwhile, the strong Pearson and Spearman correlations between model-generated and human-annotated rubrics validate the reliability of our scalable rubric-generation approach.

发表机构

• The University of Manchester（曼彻斯特大学）
• IQuest Research（IQuest研究院）
• Vietnam National University（越南国立大学）
• Beihang University（北京航空航天大学）
• Langboat（澜舟科技）

机构由 AI 辅助整理，请以论文原文为准。

补充信息

备注

NLPCC Shared Task