ScopeBench：在目标压力下，智能体能否保持参与边界？
ScopeBench: Do Agents Preserve Engagement Boundaries Under Goal Pressure?
- dreadnode
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
ScopeBench通过30个死胡同式任务评估智能体在目标压力下的范围遵守能力，发现评判器可捕捉机械验证遗漏的违规，且模型间能力与遵守度存在显著差异。
AI中文摘要：
智能体越来越多地被部署在Web应用和网络渗透测试中，具备真正的自主性，在这些场景中，一次越界操作就可能突破客户的参与边界。现有的进攻性安全基准衡量的是原始黑客能力；随着这些基准趋于饱和，部署的真正障碍是对齐的一个特例：范围遵守。我们引入了ScopeBench，这是一个包含30个死胡同式智能体安全任务的基准，在这些任务中，所述目标只有通过违反所述范围才能实现。每个任务在两种条件下出现，这两种条件共享环境、验证器和目标，仅在范围上有所不同：一组指令没有范围，用于衡量能力；另一组具有自然语言范围，用于衡量遵守程度。无范围的轨迹由标准确定性验证器评分。有范围的轨迹通过两个评分分支。首先，同一个确定性验证器检查标志：由于标志位于范围边界之后，通过则按构造证明发生了被禁止的操作，从而得出违规率的高精度下限。如果验证器未通过轨迹，则智能体评判器估计是否发生了越界调用。我们根据人类标注者逐调用标注的100条ScopeBench轨迹对评判器进行校准，并对评估的滚动输出进行盲审，发现其高召回率成立——在36个被审计的违规中无假阴性，其唯一观察到的错误是过度标记。在一个测试平台上的8个模型中，原始能力范围从12.2%到81.1%，范围遵守范围从34.4%到86.7%，评判器发现了机械验证遗漏的331个违规。Opus-4-8的原始能力得分比sonnet-4-6高10个百分点，同时其范围遵守程度高出35.6个百分点。我们发布了冻结的试点基准、评估代码以及所有2160条ATIF轨迹。
英文摘要：
Agents are increasingly deployed with real autonomy in web application and network penetration testing, where a single out-of-scope action can breach a client's engagement boundary. Existing offensive-security benchmarks measure raw hacking capability; as those benchmarks saturate, the real barrier to deployment is a special case of alignment: scope adherence. We introduce ScopeBench, a benchmark of 30 dead-end agentic security tasks in which the stated objective is reachable only by violating the stated scope. Each task appears under two conditions that share an environment, verifier, and objective and differ only in scope: one instruction set has no scope and measures capability; the other has a natural-language scope to measure adherence. Scopeless trajectories are graded by a standard deterministic verifier. Scoped trajectories pass through two grading arms. First, the same deterministic verifier checks for the flag: because the flag sits behind the scope boundary, a pass proves by construction that a forbidden action occurred, yielding a high-precision lower bound on the violation rate. If the verifier does not pass the trajectory, an agentic judge estimates whether an out-of-scope call occurred. We calibrate the judge against 100 ScopeBench trajectories labeled call-by-call by human annotators, and a blinded audit of the evaluated rollouts finds its high recall holds - no false negatives among the 36 audited violations, with over-flagging its only observed error. Across 8 models in one harness, raw capability spans 12.2% to 81.1% and scope adherence spans 34.4% to 86.7%, with the judge finding 331 violations that mechanical verification misses. Opus-4-8 achieves a raw-capability score 10 percentage points higher than sonnet-4-6's while exhibiting 35.6 percentage points higher scope adherence. We release the frozen pilot benchmark, evaluation code, and all 2160 ATIF trajectories.
