超越攻击成功率：工具使用型人工智能代理的行动分级严重程度量表
Beyond Attack-Success Rate: Action-Graded Severity Scale for Tool-Using AI Agents
AI总结：
研究指出代理红队测试基准的二元攻击成功率不能体现行动危害。为此引入行动分级伤害准则，通过预言机和评判小组计算量表。在AgentDojo工作区测试发现该准则能揭示新情况，虽与预言机一致性高但有盲点，贡献了可用于红队日志实际行动的严重程度工具。
摘要：
代理红队测试基准以单个比特报告注入的代理是否被攻破，即攻击是否成功。我们认为这种二元攻击成功率丢弃了防御者最需要的信息，即产生的行动有多有害。我们引入了一种行动分级伤害准则，根据执行的行动是否可逆、是否越界影响另一方以及是否提升权限，在七级有序量表（L0到L6）上对代理的工具调用轨迹进行评分。我们通过两种方式计算该量表：一种是读取轨迹和攻击者既定目标的确定性预言机，另一种是阅读相同轨迹无标签描述的三名前沿语言模型评判小组。在AgentDojo工作区套件中的四个受害者模型和两种防御措施上，严重程度分级揭示了二元度量隐藏的三种情况，包括一种报告零攻击成功率但仍允许通过未过滤工具出现外部可见跨范围泄露的防御措施。评判小组与预言机的序数一致性很高（Krippendorff's alpha = 0.91），但也存在我们所描述的系统性盲点，最明显的是未能识别升级链。与之前提供伤害分类法、有害任务完成测试、执行级安全基准或严重程度感知模拟的工作不同，我们的贡献是一种可重复使用、基于追踪的严重程度工具，应用于现有红队日志中记录的实际行动。所有代码、提示和每集日志均已发布。
英文摘要：
Agentic red-teaming benchmarks report whether an injected agent was compromised as a single bit: the attack succeeded, or it did not. We argue that this binary attack-success rate discards the information a defender most needs, namely how harmful the resulting action was. We introduce an action-graded harm rubric that scores an agent's tool-call trajectory on a seven-level ordinal scale (L0 to L6) according to whether the executed action was reversible, whether it crossed scope to reach another party, and whether it expanded privilege. We compute the scale two ways: a deterministic oracle that reads the trajectory and the attacker's stated goal, and a panel of three frontier language-model judges that read a tag-free account of the same trajectory. Across four victim models and two defenses on the AgentDojo workspace suite, severity grading exposes three cases the binary metric hides, including a defense that reports a zero attack-success rate while still permitting an externally visible cross-scope leak through an unfiltered tool. The judge panel reproduces the oracle with high ordinal agreement (Krippendorff's alpha = 0.91) but shares systematic blind spots that we characterize, most notably a failure to recognize escalation chains. Unlike prior work that provides harm taxonomies, harmful-task completion tests, execution-level safety benchmarks, or severity-aware simulation, our contribution is a reusable, trace-grounded severity instrument applied to the actual actions recorded in existing red-team logs. All code, prompts, and per-episode logs are released.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
