发表机构
University of Chinese Academy of Sciences; National University of Singapore; Institute of Automation, Chinese Academy of Sciences（中国科学院大学; 新加坡国立大学; 中国科学院自动化研究所）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
针对自动装备优化导致的评估作弊问题，提出反事实装备搜索与进化（CHASE）方法，通过约束生成和挑战者搜索中和基准捷径，在保持发布基准增益的同时大幅减少有效协议变化下的增益破坏。
AI 中文摘要
可靠的智能体评估因自动装备优化而变得复杂，这种优化反复使用已发布的基准 $B_{\mathrm{rel}}$ 来引导一个提议者，该提议者围绕固定的目标智能体编辑提示、记忆、检索、工具和控制代码。任务留出法改变语义任务，但保持基准协议固定，因此一个“坏天才”提议者可以产生一个作弊装备，其在已发布基准上的增益依赖于整个基准的捷径。我们引入了反事实装备搜索与进化（CHASE），它将装备进化转化为对保持有效性的基准反事实的约束生成。在每次提议者更新后，一个挑战者搜索一个具有大增益破坏性的可执行协议变换。一个有效性防火墙检查任务语义是否被保留，而一个确认集决定该反事实是否进入有限档案。我们形式化了一个精确的捷径中和基准 $B_0$，并建立了将有限反事实档案与 $B_0$ 联系起来的统计保证，以及刻画顺序挑战者搜索的特征。我们在一个合成基准和OfficeQA上评估了CHASE，在OfficeQA上，CHASE保留了强大的已发布基准增益，同时在有效协议变化下大幅减少了增益破坏。
英文摘要
Reliable agent evaluation is complicated by automatic harness optimization, which repeatedly uses a released benchmark $B_{\mathrm{rel}}$ to guide a Proposer that edits prompts, memory, retrieval, tools, and control code around a fixed foundation model. Task holdout is commonly used to guard against harness overfitting. It varies semantic tasks but leaves the benchmark protocol fixed, so a bad genius Proposer can produce a cheating harness whose improvement over the initial harness on $B_{\mathrm{rel}}$ depends on a benchmark-wide shortcut. We introduce Counterfactual Harness Search and Evolution (CHASE), which casts harness evolution as constraint generation over valid counterfactual benchmarks. After each Proposer update, a Challenger searches for an executable protocol transformation with large gain destruction. A validity firewall checks that task semantics are preserved, while a held-out confirmation set determines whether the counterfactual enters a finite archive. We formalize an ideal shortcut-neutralized benchmark $B_0$ and establish theoretical guarantees linking finite counterfactual archives to $B_0$ and characterizing sequential Challenger search. We evaluate CHASE on Syn-Ledger and OfficeQA, where CHASE retains strong released-benchmark gains while substantially reducing gain destruction under valid protocol transformations.
Comments32 pages, 6 figures; includes references and supplementary material
