AuthGateBench：将大语言模型作为授权网关的基准测试
AuthGateBench: Benchmarking LLMs as Authorization Gate
AI总结：
AuthGateBench是首个针对良性请求下授权网关的策略引导基准，通过对比对评估LLM的上下文推理能力，实验发现环境信号可显著提升授权准确率。
摘要：
自动化审批系统通过将授权决策委托给分类器（通常是大语言模型，LLM），让编码智能体在极少人工监督下执行高风险操作。这类系统已被Claude Code和Codex等工具采用，但仍存在范围升级等失效模式，即智能体执行超出用户授权范围的操作。现有基准主要关注恶意请求，基于隐式策略评估拒绝行为。我们推出AuthGateBench，这是首个针对良性请求下授权网关的策略引导基准。我们实现了一种新颖的流水线：LLM生成具有相同操作和轨迹但请求不同的对比对，在阻止和允许之间切换标签，这使我们能够评估模型是否基于上下文而非仅操作进行推理。我们进一步研究辅助环境信号是否能帮助模型做出更好的决策。最终的基准包含1400个实例，涵盖从真实Claude Code事件中提取的5种失效模式。评估显示，随着模型过度依赖内部安全对齐，不同失效模式下的准确率下降不均；即使明确授权，模型仍会阻止大多数操作，这表明决策不稳定而非确定性；然而，回溯检索的环境信号大幅提高了授权准确率。
英文摘要：
Automated approval systems let coding agents execute high-risk actions with minimal human oversight by delegating the authorization decision to a classifier, typically a large language model (LLM). Such systems, adopted in tools like Claude Code and Codex, remain vulnerable to failure modes such as scope escalation, where the agent takes actions exceeding what the user authorized. Existing benchmarks focus mainly on malicious requests, evaluating refusal under an implicit policy. We introduce AuthGateBench, the first policy-guided benchmark for authorization gating under benign requests. We implement a novel pipeline where an LLM generates contrast pairs with identical action and trajectory but differing request, flipping the label between block and allow. This allows us to assess whether models reason over context rather than the action alone. We further examine whether auxiliary environment signals can help models make better decisions. The resulting benchmark comprises 1,400 instances spanning five failure modes drawn from real-world Claude Code incidents. Evaluations show accuracy degrades unevenly across failure modes as models over-rely on internal safety alignment, and models still block most actions even when explicitly authorized, revealing unstable rather than deterministic decisions; retrospectively retrieved environment signals, however, substantially improve authorization accuracy.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
