arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2605.01644cs.CR
收藏

迈向代理安全测量的原理性框架

Toward a Principled Framework for Agent Safety Measurement

Northeastern University(东北大学);DatologyAI

Shuyi Lin, Anshuman Suri, Alina Oprea, Cheng Tan

AI总结:

本文提出基于搜索而非采样的原理性框架,用于评估代理安全,通过BOA框架在预算内搜索轨迹空间,发现贪心和采样方法遗漏的不安全轨迹。

摘要:

LLM代理发出动作而非仅文本,一旦执行便无法撤销。然而,当前代理安全评估仅运行贪心或少量采样轨迹并报告单一安全/不安全率,忽视长尾轨迹中低概率但非可忽略的动作可能导致不安全行为。本文认为代理安全应通过搜索而非采样来测量。我们应用BOA框架,给定部署配置(模型、解码器、提示、环境、判断者、似然预算),在预算内搜索轨迹空间并报告安全分数:在配置下代理保持安全的概率。BOA在单个LLM轮次内及代理-环境交互树下进行搜索,通过批量解码/判断、前缀缓存和分块树扩展使搜索可行。在代理安全工作负载上,BOA发现贪心和采样评估遗漏的不安全轨迹。BOA还可用于对模型、防御措施和攻击进行排名,所有在相同尺度上进行,具有可控的GPU成本。

英文摘要:

LLM agents emit actions, not just text, and once taken, those actions often cannot be undone. Yet today's agent-safety evaluations run greedy or a few sampled rollouts and report a single safe/unsafe rate -- blind to the long-tail trajectories where unsafe behavior may arise from low-probability but non-negligible actions. We argue agent safety should be measured by search, not sampling. We apply BOA, a framework that, given a deployment configuration (model, decoder, prompt, environment, judger, likelihood budget), searches the in-budget trajectory space and reports a safety score: the probability the agent stays safe under the configuration. BOA searches both within a single LLM round and across the agent-environment interaction tree under a given likelihood budget, and makes search practical via batched decoding/judging, prefix caching, and chunked tree expansion. On agent-safety workloads, BOA discovers unsafe trajectories that greedy and sampled evaluations miss. BOA can additionally be used for ranking models, defenses, and attacks, all on the same scale, with manageable GPU costs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑