arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.16958cs.AIcs.LG
收藏

通过结构模板注入自动化代理劫持

Automating Agent Hijacking via Structural Template Injection

Tsinghua University \& Ant Group Hangzhou China;Tsinghua University Beijing China;Zhongguancuan Laboratory Beijing China;Tsinghua University \& Ant Group;Tsinghua University;Zhongguancuan Laboratory

Xinhao Deng, Jiaqing Wu, Miao Chen, Yue Xiao, Ke Xu, Qi Li

AI总结:

本文提出Phantom框架,通过结构模板注入自动化代理劫持,提升攻击成功率和效率,揭示结构模板劫持的严重性。

摘要:

代理劫持被OWASP视为对大型语言模型(LLM)生态系统的关键威胁,使攻击者能够通过向检索内容中注入恶意指令来操控执行。现有攻击大多依赖于手动编写的、基于语义的提示操纵,通常导致攻击成功率低且难以转移到封闭源商业模型上。在本文中,我们提出了Phantom,一个基于结构模板注入的自动化代理劫持框架,旨在针对LLM代理的基本架构机制。我们的关键见解是代理依赖特定的聊天模板令牌来分离系统、用户、助手和工具指令。通过在检索上下文中注入优化的结构化模板,我们诱导角色混淆,导致代理将注入的内容误认为是合法的用户指令或先前的工具输出。为了增强对黑盒代理的攻击可转移性,Phantom引入了新的攻击模板搜索框架。我们首先进行多级模板增强以增加结构多样性,然后训练一个模板自动编码器(TAE)将离散模板嵌入到连续、可搜索的潜在空间中。随后,我们应用贝叶斯优化来高效地识别最优对抗向量,这些向量解码后会生成高潜能的结构化模板。在Qwen、GPT和Gemini上的广泛实验表明,我们的框架在攻击成功率(ASR)和查询效率方面均显著优于现有基线。此外,我们发现超过70个现实世界商业产品中的漏洞已被供应商确认,凸显了基于结构模板的劫持的实际严重性,并为安全下一代代理系统提供了实证基础。

英文摘要:

Agent hijacking, highlighted by OWASP as a critical threat to the Large Language Model (LLM) ecosystem, enables adversaries to manipulate execution by injecting malicious instructions into retrieved content. Most existing attacks rely on manually crafted, semantics-driven prompt manipulation, which often yields low attack success rates and limited transferability to closed-source commercial models. In this paper, we propose Phantom, an automated agent hijacking framework built upon Structured Template Injection that targets the fundamental architectural mechanisms of LLM agents. Our key insight is that agents rely on specific chat template tokens to separate system, user, assistant, and tool instructions. By injecting optimized structured templates into the retrieved context, we induce role confusion and cause the agent to misinterpret the injected content as legitimate user instructions or prior tool outputs. To enhance attack transferability against black-box agents, Phantom introduces a novel attack template search framework. We first perform multi-level template augmentation to increase structural diversity and then train a Template Autoencoder (TAE) to embed discrete templates into a continuous, searchable latent space. Subsequently, we apply Bayesian optimization to efficiently identify optimal adversarial vectors that are decoded into high-potency structured templates. Extensive experiments on Qwen, GPT, and Gemini demonstrate that our framework significantly outperforms existing baselines in both Attack Success Rate (ASR) and query efficiency. Moreover, we identified over 70 vulnerabilities in real-world commercial products that have been confirmed by vendors, underscoring the practical severity of structured template-based hijacking and providing an empirical foundation for securing next-generation agentic systems.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑