Large language models (LLMs) have been widely adopted in applications such as automated content generation and even critical decision-making systems. However, the risk of prompt injection allows for potential manipulation of LLM outputs. While numerous attack methods have been documented, achieving full control over these outputs remains challenging, often requiring experienced attackers to make multiple attempts and depending heavily on the prompt context. Recent advancements in gradient-based white-box attack techniques have shown promise in tasks like jailbreaks and system prompt leaks. Our research generalizes gradient-based attacks to find a trigger that is (1) Universal: effective irrespective of the target output; (2) Context-Independent: robust across diverse prompt contexts; and (3) Precise Output: capable of manipulating LLM inputs to yield any specified output with high accuracy. We propose a novel method to efficiently discover such triggers and assess the effectiveness of the proposed attack. Furthermore, we discuss the substantial threats posed by such attacks to LLM-based applications, highlighting the potential for adversaries to taking over the decisions and actions made by AI agents.
Exploring the Universal Vulnerability of Prompt-based Learning Paradigm
Tsinghua University(清华大学);Beijing National Research Center for Information Science and Technology(北京国家信息科学与技术研究中心);Huazhong University of Science and Technology(华中科技大学);Chongqing University(重庆大学)
School of Cyber Science and Engineering, Huazhong University of Science and Technology(华中科技大学网络空间安全学院);Department of Electrical and Computer Engineering, Queen’s University(女王大学电气与计算机工程系);Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所)
Red Alarm for Pre-trained Models: Universal Vulnerability to Neuron-Level Backdoor Attacks
Tsinghua University(清华大学);Institute for Artificial Intelligence, Tsinghua University(清华大学人工智能研究院);State Key Lab on Intelligent Technology and Systems, Tsinghua University(清华大学智能技术与系统国家重点实验室);Department of Computer Science and Technology, Tsinghua University(清华大学计算机科学与技术系)