LLM Agent技能的反事实痕迹审计
Counterfactual Trace Auditing of LLM Agent Skills
AI总结：
提出反事实痕迹审计（CTA）框架，通过配对有无技能的Agent轨迹并生成结构化技能影响模式（SIP）注释，揭示技能对行为的重塑效应，弥补仅通过通过率评估的不足。
摘要：
大型语言模型Agent越来越多地配备Agent技能。当前对技能的评估方法仍然有限。大多数已部署的基准测试仅报告技能附加前后的通过率，将技能视为对Agent行为的黑盒更改。我们引入了反事实痕迹审计（CTA），这是一个衡量技能如何改变Agent行为的框架。CTA将每个带技能的Agent轨迹与同一任务上不带技能的对应轨迹配对，将两条轨迹分割成目标导向的阶段，对齐这些阶段，并输出结构化的技能影响模式（SIP）注释。这些注释描述了技能的行为效果，而不仅仅是任务结果。我们在SWE-Skills-Bench上使用Claude对49个软件工程任务实例化了CTA。由此产生的审计揭示了一个明显的评估差距。通过率平均仅变化+0.3个百分点，表明总体效果很小。然而，CTA在相同的配对轨迹中识别出522个SIP实例，表明即使在通过率几乎不变的情况下，技能也显著重塑了Agent行为。审计还分离了通过率无法检测到的几种反复出现的效果，包括字面模板复制、偏离任务的人工制品创建、过度规划和任务恢复。出现了三个发现。首先，高基线任务包含了大多数观察到的技能效果，尽管它们的通过率已经饱和，因此无法反映这些效果。其次，基线性能适中的任务显示出最大的可恢复增益，但通常以显著更高的令牌成本为代价。第三，主导的SIP类型可以通过基线桶识别：表面锚定在最高任务中最常见，边缘案例提示在中档和最低任务中最常见。这些规律将非正式的故障模式观察转化为可重复的行为测量。
英文摘要：
Large Language Model agents are increasingly augmented with agent skills. Current evaluation methods for skills remain limited. Most deployed benchmarks report only pass rate before and after a skill is attached, treating the skill as a black box change to agent behavior. We introduce Counterfactual Trace Auditing (CTA), a framework for measuring how a skill changes agent behavior. CTA pairs each with skill agent trace with a without skill counterpart on the same task, segments both traces into goal directed phases, aligns the phases, and emits structured Skill Influence Pattern (SIP) annotations. These annotations describe the behavioral effect of a skill rather than only its task outcome. We instantiate CTA on SWE-Skills-Bench with Claude across 49 software engineering tasks. The resulting audit reveals a clear evaluation gap. Pass rate changes by only +0.3 percentage points on average, suggesting little aggregate effect. Yet CTA identifies 522 SIP instances across the same paired traces, showing that the skills substantially reshape agent behavior even when pass rate is nearly unchanged. The audit also separates several recurring effects that pass rate cannot detect, including literal template copying, off task artifact creation, excess planning, and task recovery. Three findings emerge. First, high baseline tasks contain most of the observed skill effects, although their pass rate is already saturated and therefore cannot reflect those effects. Second, tasks with moderate baseline performance show the most recoverable gain, but often at substantially higher token cost. Third, the dominant SIP type can be identified by baseline bucket: surface anchoring is most common on ceiling tasks and edge-case prompting is most common on mid-range and floor tasks. These regularities turn informal failure mode observations into reproducible behavioral measurements.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
