Watch Out for Your Agents! Investigating Backdoor Threats to LLM-Based Agents
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；Center for Data Science, Peking University（北京大学数据科学中心）；Pattern Recognition Center, WeChat AI, Tencent Inc.（腾讯公司微信AI模式识别中心）
Wenkai Yang, Xiaohan Bi, Yankai Lin, Sishuo Chen, Jie Zhou, Xu Sun
在大语言模型（Large Language Models, LLMs）快速发展的推动下，基于LLM的智能体已被开发用于处理各类现实应用，包括金融、医疗、购物等领域。确保基于LLM的智能体在应用过程中的可靠性与安全性至关重要。然而，目前针对基于LLM的智能体的安全问题研究仍不充分。在本研究中，我们首次针对基于LLM的智能体面临的典型安全威胁之一——后门攻击展开调查。我们首先构建了智能体后门攻击的通用框架，随后对不同形式的智能体后门攻击进行了全面分析。具体而言，与仅能操纵用户输入和模型输出的传统LLM后门攻击相比，智能体后门攻击呈现出更多样化且更隐蔽的形式：（1）从最终攻击效果来看，智能体后门攻击者不仅可以选择操纵最终输出分布，还可以仅在中间推理步骤引入恶意行为，同时保持最终输出正确。（2）此外，前者（操纵最终输出的攻击类别）可根据触发器位置分为两个子类，后门触发器既可以隐藏在用户查询中，也可以出现在外部环境返回的中间观测结果中。我们在网页购物和工具使用两个典型智能体任务上实现了上述各类智能体后门攻击变体。大量实验表明，基于LLM的智能体极易遭受后门攻击，且此类后门漏洞无法通过当前的文本后门防御算法轻易缓解。这表明亟需进一步研究开发针对基于LLM的智能体后门攻击的定向防御方法。警告：本文可能包含有偏见的内容。
英文摘要：
Driven by the rapid development of Large Language Models (LLMs), LLM-based agents have been developed to handle various real-world applications, including finance, healthcare, and shopping, etc. It is crucial to ensure the reliability and security of LLM-based agents during applications. However, the safety issues of LLM-based agents are currently under-explored. In this work, we take the first step to investigate one of the typical safety threats, backdoor attack, to LLM-based agents. We first formulate a general framework of agent backdoor attacks, then we present a thorough analysis of different forms of agent backdoor attacks. Specifically, compared with traditional backdoor attacks on LLMs that are only able to manipulate the user inputs and model outputs, agent backdoor attacks exhibit more diverse and covert forms: (1) From the perspective of the final attacking outcomes, the agent backdoor attacker can not only choose to manipulate the final output distribution, but also introduce the malicious behavior in an intermediate reasoning step only, while keeping the final output correct. (2) Furthermore, the former category can be divided into two subcategories based on trigger locations, in which the backdoor trigger can either be hidden in the user query or appear in an intermediate observation returned by the external environment. We implement the above variations of agent backdoor attacks on two typical agent tasks including web shopping and tool utilization. Extensive experiments show that LLM-based agents suffer severely from backdoor attacks and such backdoor vulnerability cannot be easily mitigated by current textual backdoor defense algorithms. This indicates an urgent need for further research on the development of targeted defenses against backdoor attacks on LLM-based agents. Warning: This paper may contain biased content.
Backdoor Collapse: Eliminating Unknown Threats via Known Backdoor Aggregation in Language Models
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；University of Science and Technology of China（中国科学技术大学）；United Arab Emirates University（阿联酋大学）；PayPal Inc（PayPal公司）；Walmart Labs（沃尔玛实验室）；Squirrel Ai Learning（Squirrel Ai学习）；Nanyang Technological University（南洋理工大学）
BadPromptFL: A Novel Backdoor Threat to Prompt-based Federated Learning in Multimodal Models
School of Information and Communication Engineering, Dalian University of Technology（信息与通信工程学院，大连理工大学）；School of Computer Science and Technology, Anhui University（计算机科学与技术学院，安徽大学）；New Laboratory of Pattern Recognition (NLPR) State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS) Institute of Automation, Chinese Academy of Sciences (CASIA)（模式识别新实验室（NLPR）多模态人工智能系统国家重点实验室（MAIS）自动化研究所，中国科学院（CASIA））
Proactive Adversarial Defense: Harnessing Prompt Tuning in Vision-Language Models to Detect Unseen Backdoored Images
University of West Florida（西佛罗里达大学）；Florida Institute For Human and Machine Cognition (IHMC)（佛罗里达人机认知研究所）；Center for Cybersecurity, University of West Florida（西佛罗里达大学网络安全中心）
School of Computer Science and Engineering, Changchun University of Technology（长春工业大学计算机科学与工程学院）；AI Research Institute, Changchun University of Technology（长春工业大学人工智能研究院）；School of Computer Science and Technology, Jilin University（吉林大学计算机科学与技术学院）
Watch Out! Simple Horizontal Class Backdoor Can Trivially Evade Defense
CSIRO’s Data61（联邦科学与工业研究组织数据61研究所）；University of Technology Sydney（悉尼科技大学）；The University of Western Australia（西澳大学）；Fudan University（复旦大学）；Nanjing University of Science and Technology（南京理工大学）；The University of Adelaide（阿德莱德大学）
Beyond Traditional Threats: A Persistent Backdoor Attack on Federated Learning
College of Computer Science and Technology, Harbin Engineering University（哈尔滨工程大学计算机科学与技术学院）；Southampton Ocean Engineering Joint Institute, Harbin Engineering University（哈尔滨工程大学南安普顿海洋工程联合学院）
Untargeted Backdoor Attack against Object Detection
Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Research Center of Artificial Intelligence, Peng Cheng Laboratory（鹏城实验室人工智能研究中心）
Towards A Critical Evaluation of Robustness for Deep Learning Backdoor Countermeasures
Nanjing University of Science and Technology（南京理工大学）；The University of Adelaide（阿德莱德大学）；Data61, CSIRO（联邦科学与工业研究组织数据61研究所）
由于深度学习（DL）后门攻击已被发现是最阴险的对抗攻击之一，因此已经开发了一些对策，并在其各自的威胁模型中定义了某些假设。然而，这些对策的鲁棒性被无意中忽略，这可能会带来严重的后果，例如，对策可能被误用并导致后门检测的错误暗示。这是第一次，我们批判性地研究了现有后门对策的鲁棒性，最初关注的是三个有影响力的模型检查，即神经清洗（S& P '19），ABS（CCS'19）和MNTD（S& P '21）。虽然这三种对策声称它们在各自的威胁模型下工作良好，但它们具有固有的未探索的非鲁棒性情况，这取决于给定的任务，模型架构，数据集和防御超参数等因素，这些因素是\textit{not even rooted from delicate adaptive attacks}。我们演示了如何通过简单地改变上述因素来绕过它们，并与它们各自的威胁模型保持一致。特别地，对于每一个防御，形式证明或实证研究被用来揭示其两个非鲁棒的情况下，它是不作为强大的，因为它声称或期望，特别是最近的MNTD。这项工作突出了彻底评估后门对策的鲁棒性，以避免其误导性的安全影响，在未知的非强大的情况下的必要性。
BADTV: Unveiling Backdoor Threats in Third-Party Task Vectors
National Yang Ming Chiao Tung University（国立阳明交通大学）；University of California, Berkeley（加州大学伯克利分校）；RIKEN（理化学研究所）；National Chengchi University（国立成功大学）；CISPA Helmholtz Center for Information Security（CISPA 河岸信息安全中心）；Institute of Science Tokyo（东京科学研究所）
Zhejiang University（浙江大学）；Binjiang Institute of Zhejiang University（浙江大学滨江研究院）；Wuhan University（武汉大学）；Huawei International, Singapore（华为国际）；Pennsylvania State University（宾夕法尼亚州立大学）
Backdoor Vectors: a Task Arithmetic View on Backdoor Attacks and Defenses
Warsaw University of Technology（华沙技术大学）；NASK National Research Institute（国家研究机构NASK）；IDEAS NCBR；IDEAS Research Institute（IDEAS研究所）；Computer Vision Center, Universitat Autonoma de Barcelona（巴塞罗那自治大学计算机视觉中心）
