arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2606.06324cs.SEcs.MA
收藏

从失败轨迹到可靠的LLM代理:诊断与修复框架缺陷

From Failed Trajectories to Reliable LLM Agents: Diagnosing and Repairing Harness Flaws

State Key Laboratory of Complex System Modeling and Simulation Technology(复杂系统建模与仿真技术国家重点实验室);Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所);University of Chinese Academy of Sciences(中国科学院大学);School of Computer Science and Technology, Tianjin University(天津大学计算机科学与技术学院)

Mengzhuo Chen, Junjie Wang, Zhe Liu, Yawen Wang, Haiming Zheng, Qing Wang

AI总结:

提出HarnessFix框架,通过轨迹引导诊断代理失败并修复执行框架,在多个基准上提升15.2%-50.0%性能。

摘要:

基于LLM的代理越来越依赖提供执行环境、工具接口、上下文、生命周期编排、可观测性、验证和治理的框架。现有的自我改进代理和自动框架演化方法主要通过运行时监督、提示优化、工作流搜索或基于最终结果的框架修改来改进代理。然而,它们通常无法诊断失败轨迹中责任证据所在的位置,以及哪个框架层导致了不可靠行为,从而导致广泛、间接或范围不当的更改。本文提出HarnessFix,一个用于诊断代理失败和修复代理框架的轨迹引导框架。HarnessFix将原始执行轨迹和框架代码编译成框架感知轨迹中间表示(HTIR),该表示标准化碎片化的轨迹证据并捕获步骤级来源和控制流关系。然后,它将失败归因于责任轨迹步骤和框架层,将重复诊断合并为可操作的缺陷记录,并将其映射到范围限定的修复操作符。最后,HarnessFix在缺陷特定的修复规范下生成并验证框架补丁,以减少目标缺陷而不引入不可接受的回归。我们在SWE-Bench Verified、Terminal-Bench 2.0 Verified、GAIA和AppWorld上评估HarnessFix。在这些基准上,HarnessFix将初始框架的留出测试性能提升了15.2%-50.0%,优于人类设计和自我演化的基线,并揭示了跨ETCLOVG层的重复框架缺陷模式。

英文摘要:

LLM agents increasingly rely on agent harness: the runtime infrastructure around the base model that defines execution environments, tool interfaces, context, lifecycle orchestration, observability, verification, and governance. Existing self-improving agents and automatic harness evolution methods mainly improve agents through runtime supervision, prompt optimization, workflow search, or harness modification based on final outcomes. However, they often fail to diagnose where the responsible evidence lies in failed trajectories and which harness implementation mechanism causes the unreliable behavior, resulting in broad, indirect, or poorly scoped changes. This paper proposes HarnessFix, a trace-grounded and diagnosis-driven framework for repairing agent harnesses. HarnessFix compiles raw execution traces and harness artifacts into a Harness-aware Trace Intermediate Representation (HTIR), which normalizes fragmented trajectory evidence, captures step-level data-flow and control-flow relations, and aligns runtime steps with the harness artifacts that shape their behavior. It then attributes failures to responsible steps and harness artifacts, and consolidates recurring diagnoses into repair-oriented flaw records. Finally, HarnessFix maps these records to scoped repair operators, generates patches under flaw-specific repair specifications, and accepts them through regression-aware validation. We evaluate HarnessFix on four popular benchmarks, and results show that it improves the performance over the initial harnesses by 6.3% to 18.4%, significantly outperforming human-designed and self-evolution baselines. HarnessFix highlights the value of treating failed trajectories not only as feedback signals, but also as structured evidence for diagnosing and repairing the harness mechanisms behind agent failures.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑