arXiv 2605.061362026-05cs.SEcs.AI

BUILD-AND-FIND: 一种考虑努力的协议用于评估代理管理的代码库

Build-and-Find: An Effort-Aware Protocol for Evaluating Agent-Managed Codebases

National Yang Ming Chiao Tung University

Jhen-Ke Lin

备注：: 36 pages, 8 figures. Accepted at NeurIPS 2026. Code: https://github.com/JacobLinCool/Build-and-Find

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

该研究提出BUILD-AND-FIND协议，用于评估下游代理从生成代码库中恢复意图选择的能力及所需检查努力，通过分离行为正确性与恢复过程，评估准确性、稳定性、重复性、覆盖范围和努力程度。

摘要：

大多数代码代理基准测试询问生成的代码是否正确。尽管这仍然重要，但仓库级别的工程越来越多地由代理管理：一个代理编写仓库，后来的代理会检查、审计或扩展它作为工作上下文。在这种情况下，生成的仓库不仅是任务的答案，也是未来工作的沟通 artifacts。即使强代理几乎满足可见行为目标，仓库在暴露意图行为和设计选择方面的清晰度可能不同。我们引入BUILD-AND-FIND协议，用于评估下游代理是否能从生成的仓库中恢复这些意图选择，以及恢复所需多少检查努力。对于每个任务，构建者看到隐藏的仓库规范并创建代码库；查找者只能看到代码库和一个带有规范追溯的多选问题库。该协议将行为正确性与 artifact 侧的恢复分离，并报告恢复准确性、稳定性、重复性、实现覆盖范围和检查努力。准确性与稳定性作为闸门：只有在恢复成功可靠时，努力才被解释。在从相同意图可以恢复的 artifacts 中，相同查找者更低的努力表明该 artifact 使该意图更容易定位。仅问题和仅规范的控制量度通用先验和规范访问，而审计将遗漏的声明与查找者失败分离，并检查正确答案是否引用 artifact 证据。在发布的高优先级任务包中，恢复准确性接近饱和，因此检查努力和查找者特定效应提供了主要的面板局部比较。

英文摘要：

Most coding-agent benchmarks ask whether generated code behaves correctly. That remains essential, but repository-level engineering is increasingly agent-managed: one agent writes a repository, and later agents inspect, audit, or extend it as working context. In that setting, a generated repository is not only an answer to a task but also a communication artifact for future work. Even when strong agents nearly satisfy the visible behavioral objective, repositories can differ in how clearly they expose the intended behavior and design choices behind that behavior. We introduce Build-and-Find to evaluate whether downstream agents can efficiently recover intended choices from generated repositories. For each task, a builder creates a codebase from a hidden specification; a finder receives only the codebase and a specification-traced question bank. The protocol separates behavioral correctness from artifact-side recovery and reports recovery accuracy, repeatability, implementation coverage, and inspection effort. Evaluation proceeds in two stages: the first uses recovery accuracy and stability as gates to establish that finders can reliably recover the intended choices; the second compares inspection effort by the same finder across artifacts that pass these gates and allow the same intent to be recovered. Lower effort suggests that the artifact makes that intent easier to locate. We find that several frontier models approach saturation in intent recovery on our task pack, yet second-stage inspection effort reveals differences among their generated artifacts, providing a dimension beyond accuracy for comparing artifact legibility. We also audit artifacts to distinguish unimplemented specification claims from finder failures and check whether correct answers are supported by artifact evidence. We release the harness and benchmark data for reproducibility and extension.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

AI总结：

摘要：

英文摘要：

相关论文