arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.28425cs.CRcs.AI
收藏

攻击者离开后:对大语言模型智能体的遗留攻击进行基准测试

After the Attacker Leaves: Benchmarking Carryover Attacks on LLM Agents

MAIS, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, UCAS(中国科学院大学人工智能学院);Zhongguancun Academy(中关村研究院);SenseTime Research(商汤科技研究院);Zhongguancun Institute of Artificial Intelligence(中关村人工智能研究院)

Zhiyue Xu, Xukai Wang, Zhiyuan Huang, Mingjie Zhan, Jian Zhao, Xu-Yao Zhang

AI总结:

本研究提出 CarryoverBench 基准测试遗留攻击,发现前沿 LLM 智能体的植入有效载荷可跨会话重新触发且衰减微弱,触发阶段防护效果有限,需运行时与提示级结合防御。

摘要:

现代大语言模型(LLM)智能体通过两个渠道在会话间持久保存状态:一是静默附加到系统提示词的长期记忆文件,二是无需用户输入即可唤醒智能体的定时任务。攻击者在任一渠道实施的一次性间接提示注入,可在攻击者离开后跨会话重新触发,这类威胁我们称为遗留攻击。现有的提示注入基准仅评估恶意内容首次被观察到的会话中的攻击,未对植入的有效载荷后续重新触发的时机和频率的管控进行测量。我们提出 CarryoverBench:涵盖 Instant(同轮次间接提示注入)、Autonomous(心跳调度)和 Latent(记忆复现)三个触发渠道的12个个人助手场景及101项攻击任务。基于规则的验证器将植入和触发作为独立阶段评分。在五个前沿模型上,植入的有效载荷在五轮中重新触发且无明显衰减;在植入成功的条件下,触发阶段通常几乎无法提供额外保护。将 Latent 渠道移植到第二个运行时 Hermes 后,该效应有所减弱但未消除,这表明遗留攻击防御除了提示级干预外,还需要运行时层面的控制。

英文摘要:

Modern LLM agents persist state across sessions via two channels: long-lived memory files silently prepended to the system prompt, and scheduled tasks waking the agent without user input. A one-time indirect prompt injection committed to either channel can re-fire across sessions long after the attacker leaves, threats we term carryover attacks. Existing prompt-injection benchmarks evaluate attacks only in the session where malicious content is first observed, leaving the gate governing when and how often a planted payload later re-fires unmeasured. We present CarryoverBench: 12 personal-assistant scenarios and 101 attack tasks across three trigger channels: Instant, same-turn IPI; Autonomous, heartbeat-scheduled; and Latent, memory-resurfaced. A rule-based verifier scores planting and triggering as independent stages. Across five frontier models, planted payloads re-fire without measurable decay over five rounds; conditional on successful planting, the trigger stage often provides little additional protection. Porting the Latent channel to a second runtime, Hermes, attenuates but does not eliminate the effect, suggesting that carryover defense requires runtime-layer controls in addition to prompt-level interventions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑