arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2610.34577cs.AI
收藏

重新思考智能体管控框架中的验证机制

Rethinking Verification in Agent Harnesses

Cornell University(康奈尔大学);Alibaba Group(阿里巴巴集团)

Zhengxin Zhang, Erkang Zhu, Yin Lin, Claire Cardie, Bolin Ding, Jingren Zhou

AI总结:

本研究提出智能体管控框架的验证框架,对比17种设计发现独立审查在准确率-成本权衡上更优,为验证设计提供实证指导并支持其作为测试时扩展维度。

摘要:

智能体管控框架(Agent harnesses)使大语言模型能够通过工具行动、观察结果并迭代完成任务目标,验证机制用于检查和修正智能体的工作以提升输出质量,但目前关于管控框架应如何组织该工作的指导有限。本研究将验证视为测试时扩展的一个维度,研究干预时机、检查主体及检查如何促成修正,提出的框架涵盖智能体生命周期的四个层级:初始指令、回合中提醒、完成关卡、独立审查。我们在Terminal-Bench(TB)和SWE-bench Verified(SWE)上对比了7种前沿编码模型的17种设计,并引入VisualBench用于专家评估视觉与交互产物。初始指令和执行时提醒在编码基准上产生适度的、依赖模型的平均增益,未显著增加平均输出token使用量;完成关卡可产生更大增益但可能显著提升成本;由批评者进行的独立审查提供了更优的准确率-成本权衡,在TB上平均准确率提升5.9-9.5个百分点、SWE上提升3.9-6.6个百分点,成本分别为基线的1.13-1.25倍和1.00-1.04倍;在VisualBench上,独立审查在其4种设计中较基线实现13.2-23.8点的净偏好,超过评估的L0-L2设计;额外审查轮次或独立批评者可进一步将TB平均准确率提升最多3.5个百分点。这些结果为智能体管控框架中的验证设计提供了实证指导,并支持将独立审查作为测试时扩展的实用维度。

英文摘要:

Agent harnesses enable large language models to act through tools, observe the results, and iteratively work toward a task goal. Verification checks and revises agents' work to improve output quality. Yet there is limited guidance on how harnesses should organize this effort. We study verification as an axis of test-time scaling by examining when to intervene, who checks the work, and how checking leads to revision. Our framework spans four layers of the agent lifecycle: initial instructions, mid-turn reminders, completion gates, and independent review. We compare 17 designs across seven frontier coding models on Terminal-Bench (TB) and SWE-bench Verified (SWE), and introduce VisualBench for expert evaluation of visual and interactive artifacts. Initial instructions and execution-time reminders yield modest, model-dependent average gains on the coding benchmarks, without substantially increasing average output token use. Completion gates can yield larger gains but may substantially increase cost. Independent review by a critic offers a more favorable accuracy-cost tradeoff, improving average accuracy by 5.9-9.5 percentage points on TB and 3.9-6.6 on SWE at 1.13-1.25 and 1.00-1.04 times baseline cost, respectively. On VisualBench, independent review achieves net preferences of 13.2-23.8 points over the baseline across its four designs, exceeding the evaluated L0-L2 designs. Additional review rounds or independent critics further improve average TB accuracy by up to 3.5 percentage points. Together, these results provide empirical guidance for verification design in agent harnesses and support independent review as a practical axis of test-time scaling.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑