arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.23950cs.AIcs.SE
收藏

停止比较 LLM Agent 而不公开其执行框架

Stop Comparing LLM Agents Without Disclosing the Harness

Tulane University(Tulane 大学);Rutgers University(Rutgers 大学);Independent Researcher(独立研究者);Virginia Tech(弗吉尼亚理工大学)

Yunbei Zhang, Janet Wang, Yingqiang Ge, Weijie Xu, Jihun Hamm, Chandan K. Reddy

AI总结:

本文论证在长周期任务中,Agent 执行框架(Harness)比底层模型更能决定性能,并提出框架感知的评估标准与方差分解协议。

摘要:

这篇立场论文认为,对于在具有可比前沿能力的模型上评估的长周期任务,Agent 执行框架(即围绕语言模型管理上下文构建、工具交互、编排和验证的基础设施层)通常比其包装的模型更能决定 Agent 性能。我们形式化并辩护了绑定约束论题:在此情况下,性能方差更多地由框架配置而非模型选择决定,当前评估协议因此系统性地将框架层面的提升错误归因于模型改进。我们从三个方面支持这一论点。首先,控制论形式化将框架视为闭环动态系统的控制器,LLM 为其管理的随机策略,这解释了为什么小的框架变化可以产生超过替换模型所带来的性能变化。其次,已发表的基准测试、行业部署以及受控方差分解表明,框架引起的方差可能显著超过模型引起的方差,包括模型排名反转的情况。第三,我们提出了一个框架感知的评估框架,包含披露标准和方差分解协议。在框架规范被公开之前,长周期 Agent 的排行榜比较应被视为不完整且可能具有误导性。

英文摘要:

This position paper argues that, for long-horizon tasks evaluated across models with comparable frontier capability, the agent execution harness, namely the infrastructure layer that governs context construction, tool interaction, orchestration, and verification around a language model, is often a stronger determinant of agent performance than the model it wraps. We formalize and defend the Binding Constraint Thesis: in this regime, performance variance is governed more by harness configuration than by model choice, and current evaluation protocols therefore systematically misattribute harness-level gains to model improvements. We support this thesis along three lines. First, a control-theoretic formalization treats the harness as the controller of a closed-loop dynamical system and the LLM as the stochastic policy it governs, which explains why small harness changes can produce performance shifts that exceed those obtained by substituting one model for another. Second, published benchmarks, industry deployments, and a controlled variance decomposition show that harness-induced variance can substantially exceed model-induced variance, including cases of model ranking reversal. Third, we propose a harness-aware evaluation framework with a disclosure standard and a variance decomposition protocol. Until harness specifications are disclosed, leaderboard comparisons for long-horizon agents should be treated as incomplete and potentially misleading.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑