ReplicatorBench：用于社会科学和行为科学中可复制性评估的LLM代理基准测试
ReplicatorBench: Benchmarking LLM Agents for Replicability in Social and Behavioral Sciences
AI总结：
本文提出ReplicatorBench，一个端到端的基准测试，用于评估AI代理在可复制性研究中的能力，涵盖数据提取、实验设计与结果解释三个阶段，并开发了ReplicatorAgent框架以评估不同LLM和编程语言的性能。
摘要：
文献中对AI代理在自动评估科学论文方面的兴趣日益增加。现有基准主要关注任务的计算方面，测试代理在拥有代码和数据时复制研究结果的能力。这种设置虽然基础，但（1）未能捕捉到新数据复制与再生产不一致的可用性，（2）缺乏地面真实多样性，仅关注可重复论文，从而无法评估代理识别不可复制研究的能力。此外，大多数基准仅评估结果而非复制过程。为此，我们引入ReplicatorBench，一个端到端的基准测试，包括社会科学和行为科学中的人类验证可复制和不可复制的研究声明，用于评估AI代理在三个阶段：（1）复制数据的提取和检索；（2）计算实验的设计和执行；（3）结果的解释，允许测试AI代理模仿人类复制者在现实世界中的能力。为了建立AI代理能力的基线，我们开发了ReplicatorAgent，一个配备必要工具（如网络搜索和迭代交互沙盒环境）的代理框架，以完成ReplicatorBench任务。我们评估了ReplicatorAgent在四个基础大语言模型（LLM）以及不同编程语言设计和代码访问级别的性能。我们的发现表明，尽管当前LLM代理能够有效设计和执行计算实验，但它们在检索必要资源（如新数据）以复制声明方面存在困难。所有代码和数据均在https://github.com/CenterForOpenScience/llm-benchmarking公开。
英文摘要：
The literature has witnessed an emerging interest in AI agents for automated assessment of scientific papers. Existing benchmarks focus primarily on the computational aspect of this task, testing agents' ability to reproduce or replicate research outcomes when having access to the code and data. This setting, while foundational, (1) fails to capture the inconsistent availability of new data for replication as opposed to reproduction, and (2) lacks ground-truth diversity by focusing only on reproducible papers, thereby failing to evaluate an agent's ability to identify non-replicable research. Furthermore, most benchmarks only evaluate outcomes rather than the replication process. In response, we introduce ReplicatorBench, an end-to-end benchmark, including human-verified replicable and non-replicable research claims in social and behavioral sciences for evaluating AI agents in research replication across three stages: (1) extraction and retrieval of replication data; (2) design and execution of computational experiments; and (3) interpretation of results, allowing a test of AI agents' capability to mimic the activities of human replicators in real world. To set a baseline of AI agents' capability, we develop ReplicatorAgent, an agentic framework equipped with necessary tools, like web search and iterative interaction with sandboxed environments, to accomplish tasks in ReplicatorBench. We evaluate ReplicatorAgent across four underlying large language models (LLMs), as well as different design choices of programming language and levels of code access. Our findings reveal that while current LLM agents are capable of effectively designing and executing computational experiments, they struggle with retrieving resources, such as new data, necessary to replicate a claim. All code and data are publicly available at https://github.com/CenterForOpenScience/llm-benchmarking.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
