Rollout Cards: 代理研究的可重复性标准
Rollout Cards: A Reproducibility Standard for Agent Research
AI总结：
本文提出Rollout Cards作为代理研究的可重复性标准，通过保留rollout记录并声明报告规则，解决评估结果不可复现的问题。
摘要：
长期影响机器学习和强化学习的可重复性问题现在在代理研究中显现：论文通过报告分数比较系统，但留下难以检查的rollout记录。对于代理任务，这很重要，因为相同行为在评估选择不同rollout部分或应用不同报告规则时可能获得不同报告分数。在对50个流行的训练和评估仓库的结构审计中，我们发现没有报告伴随头条分数的失败运行、错误或跳过数量。我们还记录了37种情况，报告规则可以改变任务成功率、成本/令牌会计或定时测量，有时影响深远。我们将rollout记录而非报告分数视为代理研究的可重复性单位。我们引入rollout cards：出版物捆绑，保留rollout记录并声明报告规则、视图和drops清单。我们通过两种设置验证rollout cards。首先，四个部分公开发布在工具安全、多代理系统、定理证明和搜索中，使我们能够计算其原始报告未包含的分析。其次，重新评分保留的基准输出在简答、代码生成和工具使用任务中显示，仅改变报告规则可以改变报告分数20.9个绝对百分点，有时翻转前沿模型的排名。我们发布了一个参考实现，整合到Ergon，一个开源强化学习gym中，并公开发布Ergon生成的rollout-card导出，涵盖工具使用、软件工程、网络交互、多代理协调、安全和搜索等基准，以支持未来研究。
英文摘要：
Reproducibility problems that have long affected machine learning and reinforcement learning are now surfacing in agent research: papers compare systems by reported scores while leaving the rollout records behind those scores difficult to inspect. For agentic tasks, this matters because the same behaviour can receive different reported scores when evaluations select different parts of a rollout or apply different reporting rules. In a structured audit of 50 popular training and evaluation repositories, we find that none report how many runs failed, errored, or were skipped alongside headline scores. We also document 37 cases where reporting rules can change task-success rates, cost/token accounting, or timing measurements for fixed evidence, sometimes dramatically. We treat rollout records, not reported scores, as the unit of reproducibility for agent research. We introduce rollout cards: publication bundles that preserve the rollout record and declare the views, reporting rules, and drops manifests behind reported scores. We validate rollout cards in two settings. First, four partial public releases in tool safety, multi-agent systems, theorem proving, and search let us compute analyses their original reports did not include. Second, re-grading preserved benchmark outputs across short-answer, code-generation, and tool-use tasks shows that changing only the reporting rule can change reported scores by 20.9 absolute percentage points and, in some cases, invert rankings of frontier models. We release a reference implementation integrated into Ergon, an open-source reinforcement learning gym, and publicly publish Ergon-produced rollout-card exports for benchmarks spanning tool use, software engineering, web interaction, multi-agent coordination, safety, and search to support future research.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
