OpenSec: 在对抗性证据下评估事件响应代理的校准
OpenSec: Measuring Incident Response Agent Calibration Under Adversarial Evidence
AI总结：
OpenSec通过双控制强化学习环境评估事件响应代理在对抗性证据下的校准能力，发现前沿模型存在过度触发问题，校准差距体现在克制而非检测。
摘要：
随着大型语言模型（LLMs）的提升，其攻击性应用也日益增强：前沿代理现在能够以不到50美元的计算成本生成有效的漏洞利用工具（Heelan, 2026）。防御性事件响应（IR）代理必须跟上步伐，但现有基准测试将动作执行与正确执行混为一谈，当代理处理对抗性证据时隐藏了校准失败。我们引入了OpenSec，一个双控制强化学习（RL）环境，它在现实的提示注入场景下评估IR代理，采用基于执行的评分：首次遏制时间（TTFC）、证据门控动作率（EGAR）、影响范围和每层注入违规率。在40个标准层级的回合上评估四个前沿模型，我们发现了一致的过度触发现象：GPT-5.2在100%的回合中执行遏制，但有82.5%的假阳性率，在收集到足够证据之前在第4步就行动。Claude Sonnet 4.5显示部分校准（62.5%遏制，45%假阳性，TTFC为10.6），表明校准在前沿模型中并不一致。所有模型在行动时都能正确识别真实威胁；校准差距不在于检测，而在于克制。代码可在https://github.com/jbarnes850/opensec-env获取。
英文摘要：
As large language models (LLMs) improve, so do their offensive applications: frontier agents now generate working exploits for under $50 in compute (Heelan, 2026). Defensive incident response (IR) agents must keep pace, but existing benchmarks conflate action execution with correct execution, hiding calibration failures when agents process adversarial evidence. We introduce OpenSec, a dual-control reinforcement learning (RL) environment that evaluates IR agents under realistic prompt injection scenarios with execution-based scoring: time-to-first-containment (TTFC), evidence-gated action rate (EGAR), blast radius, and per-tier injection violation rates. Evaluating four frontier models on 40 standard-tier episodes each, we find consistent over-triggering: GPT-5.2 executes containment in 100% of episodes with 82.5% false positive rate, acting at step 4 before gathering sufficient evidence. Claude Sonnet 4.5 shows partial calibration (62.5% containment, 45% FP, TTFC of 10.6), suggesting calibration is not reliably present across frontier models. All models correctly identify the ground-truth threat when they act; the calibration gap is not in detection but in restraint. Code available at https://github.com/jbarnes850/opensec-env.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
