测量多步语言模型智能体中工具诱导的信念差异
Measuring Harness-Induced Belief Divergence in Multi-Step LLM Agents
AI总结：
研究多步语言模型智能体中工具对其信念的影响，引入信念展开诊断，定义跨工具信念差异并分解，通过实验表明工具设计是智能体评估中的实验变量。
摘要：
软件代理基准通常报告代理是否解决任务，但代理通过控制其所见、可采取的行动、修复的故障、验证的状态和记录的证据的工具来达到该结果。我们表明，即使任务、环境和基础语言模型固定，该工具也会改变代理的多步信念。我们引入了一种信念展开诊断，它在替代工具下引出关于进展、风险、可恢复性、约束、故障模式、不确定性、未来成功、修复成本和下一个行动的结构化K步轨迹。我们定义了一种跨工具信念差异，并将其分解为即时接口转移的到达项和与视野相关的信念变化的增长项。在受控编码任务和公共基准压力测试中，被阻止的行动、压缩的修复、选择性验证和成本感知的证据修剪通常在保持终端成功的同时改变驱动后续决策的信念。我们进一步引入了BIWM，一种无训练协议，它规范观察、记录被审查的分支、扩展修复轨迹、记录验证掩码、在影子中执行有风险的分支，并在不同工具视图之间对齐信念轨迹。结果表明，工具设计是代理评估中的一个实验变量，而不是一个实现细节。我们的代码可在此https URL上获取。
英文摘要：
Software agents are becoming increasingly important in modern coding workflows, where they solve complex tasks through multi-step interaction with code repositories. A central component of these systems is the execution harness, which governs how the underlying LLM observes, acts on, and verifies its environment. We show that the harness can in turn shape the agent's evolving beliefs about the environment, even when the task, environment, and base LLM remain fixed. To reveal this effect, we introduce a belief-rollout diagnostic that traces how an agent's anticipated world state and future decisions evolve under alternative harnesses. We further separate immediate divergence introduced by the interface from divergence that develops over longer horizons. Across controlled coding tasks include HIBench-Code, SWE-bench Verified and Terminal-Bench, we find substantial cross-harness belief divergence that is often invisible from terminal success alone and can propagate into different downstream actions. Building on this observation, we introduce \textbf{Belief-Invariant World-Modeling (BIWM)}, which exposes evidence hidden or compressed by harness execution and aligns the resulting belief trajectories across different harnesses. Our findings reveal that harness design can systematically shape how software agents understand and act on their environments, providing practical guidance for building and evaluating more reliable coding agents.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
