BenchGuard：谁守护着基准？对LLM代理基准的自动化审计
Who Guards the Benchmarks? Automated Auditing of LLM Agent Benchmarks
AI总结：
BenchGuard通过结构化LLM协议交叉验证所有基准制品，发现ScienceAgentBench中的12个问题，并在BIXBench Verified-50子集中准确识别83.3%的问题，证明自动化基准审计的有效性。
摘要：
随着基准复杂性增加，许多显而易见的代理失败并非代理本身的问题，而是基准本身的失败：规格错误、隐含假设和刚性评估脚本，这些会惩罚有效的替代方法。我们提出利用前沿LLM作为系统审计者，通过BenchGuard实现这一愿景，即首个针对任务导向型执行代理基准的自动化审计框架。BenchGuard通过结构化LLM协议交叉验证所有基准制品，可选地结合代理解决方案或执行轨迹作为额外诊断证据。在两个著名科学基准上部署后，BenchGuard发现了ScienceAgentBench中的12个作者确认的问题，包括使任务无法解决的致命错误，并在BIXBench Verified-50子集中准确匹配83.3%的专家识别问题，发现此前人类审查完全遗漏的缺陷。对50个复杂生物信息学任务的完整审计成本低于15美元，证明自动化基准审计是人类审查的实用且有价值的补充。这些发现表明AI辅助的基准开发，前沿模型不仅作为评估对象，还作为验证评估基础设施的积极参与者。
英文摘要：
As benchmarks grow in complexity, many apparent agent failures are not failures of the agent at all---they are failures of the benchmark itself: broken specifications, implicit assumptions, and rigid evaluation scripts that penalize valid alternative approaches. We propose employing frontier LLMs as systematic auditors of evaluation infrastructure, and realize this vision through BenchGuard, the first framework explicitly designed for joint cross-artifact auditing of execution-based agent benchmarks. BenchGuard cross-verifies all benchmark artifacts via structured LLM protocols, optionally incorporating agent solutions or execution traces as additional diagnostic evidence. Deployed on two prominent scientific benchmarks, BenchGuard identified 12 author-confirmed issues in ScienceAgentBench---including fatal errors rendering tasks unsolvable---and exactly matched 83.3% of expert-identified issues on the BIXBench Verified-50 subset, catching defects that prior human review missed entirely. A full audit of 50 complex bioinformatics tasks costs under USD 15, making automated benchmark auditing a practical and valuable complement to human review. A preliminary native-format audit of ProgramBench further demonstrates cross-format applicability. These findings point toward AI-assisted benchmark development, where frontier models serve not only as subjects of evaluation but as active participants in validating the evaluation infrastructure itself.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
