arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.22643cs.CL
收藏

温水煮青蛙:面向智能体安全的多轮基准测试

Boiling the Frog: A Multi-Turn Benchmark for Agentic Safety

Icaro Foundation(Icaro基金会);Sapienza University of Rome(罗马萨皮恩扎大学);Sant’Anna School of Advanced Studies(圣安娜高级研究学校);Tongji University School of Law(同济大学法学院);AIQI Consortium(AIQI联盟);Università Cattolica del Sacro Cuore(圣心大学);Piccadilly Labs(皮卡迪利实验室);VU Amsterdam(阿姆斯特丹伏伊大学);Independent(独立)
Piercosma Bisconti, Matteo Prandi, Federico Pierucci, Federico Sartore, Enrico Panai, Laura Caroli, Yue Zhu, Adam Leon Smith, Luca Nannini, Marcello Galisai, Su… 展开作者

Piercosma Bisconti, Matteo Prandi, Federico Pierucci, Federico Sartore, Enrico Panai, Laura Caroli, Yue Zhu, Adam Leon Smith, Luca Nannini, Marcello Galisai, Susanna Cifani, Francesco Giarrusso, Marcantonio Bracale Syrnikov, Daniele Nardi

AI总结:

提出一个多轮基准测试,评估工具使用AI模型在办公场景中对渐进式攻击的脆弱性,发现平均攻击成功率为44.4%。

摘要:

背景。传统的语言模型安全基准评估生成的文本:模型是否输出有毒语言、再现偏见或遵循有害指令。当模型作为智能体部署时,安全相关的对象从系统所说的内容转移到它在环境中执行的操作,仅评估模型在提示下的响应已不足以应对人工智能带来的安全挑战。近期发展出现了评估大型语言模型作为智能体的基准测试。我们对此研究方向做出贡献。方法。我们引入“温水煮青蛙”基准测试,评估部署在企业及办公环境中使用工具的AI模型是否易受渐进式攻击。每个场景以良性工作区编辑开始,随后引入一个包含风险的请求。基准测试聚焦于有状态的多轮评估:链暴露持久工作区,将风险负载置于轮次序列中的受控位置,并评估最终工件状态是否变得不安全。场景基于“温水煮青蛙”风险、AI法案附件I和附件III高风险情境以及欧盟AI法案通用人工智能行为准则(GPAI)构建的三级操作风险分类进行组织。结果。在九个模型的面板中,总体严格攻击成功率(ASR)为44.4%。模型级ASR范围从Claude Haiku 4.5的20.5%到Gemini 3.1 Flash Lite的92.9%,Seed 2.0 Lite也超过80%。对于行为准则失控场景,平均链类别级ASR达到93.3%。

英文摘要:

Background. Traditional safety benchmarks for language models evaluate generated text: whether a model outputs toxic language, reproduces bias, or follows harmful instructions. When models are deployed as agents, the safety-relevant object shifts from what the system says to what it does within an environment, and evaluating model responses under prompting is no longer sufficient to address the safety challenges posed by artificial intelligence. Recent developments have seen the rise of benchmarks that evaluate large language models as agents. We contribute to this strand of research. Approach. We introduce Boiling the Frog, a benchmark that evaluates whether tool-using AI models deployed in corporate and office settings are susceptible to incremental attacks. Each scenario begins with benign workspace edits and later introduces a risk-bearing request. The benchmark focuses on stateful multi-turn evaluation: chains expose a persistent workspace, place the risk-bearing payload at controlled positions in the turn sequence, and score whether the resulting artifact state becomes unsafe. Scenarios are organized through a three-level operational risk taxonomy grounded in the Boiling the Frog risks, the AI Act Annex I and Annex III high-risk contexts, and EU AI Act's Code of Practice on General-Purpose AI (GPAI). Results. Across a nine-model panel, aggregate strict attack success rate (ASR) is 44.4%. Model-level ASR ranges from 20.5% for Claude Haiku 4.5 to 92.9% for Gemini 3.1 Flash Lite, with Seed 2.0 Lite also above 80%. Average chain category-level ASR reaches 93.3% for Code of Practice loss-of-control scenarios.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑