超越任务完成：通过过程感知评估揭示LLM代理中的腐败成功
Beyond Task Completion: Revealing Corrupt Success in LLM Agents through Procedure-Aware Evaluation
AI总结：
本文提出过程感知评估框架，揭示LLM代理中隐藏的腐败成功，分析不同模型在任务执行中的失败模式及基准设计缺陷。
摘要：
基于大型语言模型（LLM）的代理越来越多地应用于高风险环境，但当前的基准测试主要评估任务是否完成，而不是如何完成。我们引入了过程感知评估（PAE），这是一种框架，将代理的过程形式化为结构化的观察，并揭示代理所观察、交流和执行之间的一致性关系。PAE沿着互补的轴（效用、效率、交互质量、过程完整性）评估代理，并应用多维门控，该门控可以有条理地排除腐败结果。在tau-bench上评估最先进的LLM代理，得出在轴、合规性和基准层面的发现。在轴层面，维度捕捉了非冗余的失败模式：效用掩盖可靠性差距，速度并不意味着精度，简洁性并不预测意图的遵守。在过程合规性层面，27-78%的基准报告成功是腐败成功，隐藏了交互和完整性的违规。此外，门控显著降低了Pass^4率并影响了模型排名。对腐败成功案例的分析揭示了每个模型的失败特征：GPT-5在政策、执行和意图维度上扩散错误；Kimi-K2-Thinking将78%的违规集中在政策忠实性和合规性上；而Mistral-Large-3则由忠实性失败主导。在基准层面，我们的分析揭示了基准设计中的结构性缺陷，包括任务范围差距、矛盾的奖励信号以及模拟器伪影，这些伪影会产生偶然的成功。
英文摘要：
Large Language Model (LLM)-based agents are increasingly adopted in high-stakes settings, but current benchmarks evaluate mainly whether a task was completed, not how. We introduce Procedure-Aware Evaluation (PAE), a framework that formalizes agent procedures as structured observations and exposes consistency relationships between what agents observe, communicate, and execute. PAE evaluates agents along complementary axes (Utility, Efficiency, Interaction Quality, Procedural Integrity) and applies multi-dimensional gating that categorically disqualifies corrupt outcomes. Evaluating state-of-the-art LLM agents on tau-bench yields findings at the axis, compliance, and benchmark levels. At the axis level, the dimensions capture non-redundant failure modes: utility masks reliability gaps, speed does not imply precision, and conciseness does not predict intent adherence. At the procedural compliance level, 27-78% of benchmark reported successes are corrupt successes concealing violations across interaction and integrity. Furthermore, gating substantially collapses Pass^4 rate and affects model rankings. The analysis of corrupt success cases reveals distinctive per-model failure signatures: GPT-5 spreads errors across policy, execution, and intent dimensions; Kimi-K2-Thinking concentrates 78% of violations in policy faithfulness and compliance; and Mistral-Large-3 is dominated by faithfulness failures. At the benchmark level, our analysis exposes structural flaws in the benchmark design, including task scope gaps, contradictory reward signals, and simulator artifacts that produce accidental successes.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
