Defending Large Language Models Against Attacks With Residual Stream Activation Analysis
HiddenLayer（隐层公司）；Oregon State University（俄勒冈州立大学）
Amelia Kawasaki, Andrew Davis, Houssam Abbas
备注：
Included in Proceedings of the Conference on Applied Machine Learning in Information Security (CAMLIS 2024), Arlington, Virginia, USA, October 24-25, 2024
The widespread adoption of Large Language Models (LLMs), exemplified by OpenAI's ChatGPT, brings to the forefront the imperative to defend against adversarial threats on these models. These attacks, which manipulate an LLM's output by introducing malicious inputs, undermine the model's integrity and the trust users place in its outputs. In response to this challenge, our paper presents an innovative defensive strategy, given white box access to an LLM, that harnesses residual activation analysis between transformer layers of the LLM. We apply a novel methodology for analyzing distinctive activation patterns in the residual streams for attack prompt classification. We curate multiple datasets to demonstrate how this method of classification has high accuracy across multiple types of attack scenarios, including our newly-created attack dataset. Furthermore, we enhance the model's resilience by integrating safety fine-tuning techniques for LLMs in order to measure its effect on our capability to detect attacks. The results underscore the effectiveness of our approach in enhancing the detection and mitigation of adversarial inputs, advancing the security framework within which LLMs operate.
Illinois Institute of Technology（伊利诺伊理工学院）；Rochester Institute of Technology（罗切斯特理工学院）
Deep learning models have been shown to be vulnerable to adversarial attacks. In particular, gradient-based attacks have demonstrated high success rates recently. The gradient measures how each image pixel affects the model output, which contains critical information for generating malicious perturbations. In this paper, we show that the gradients can also be exploited as a powerful weapon to defend against adversarial attacks. By using both gradient maps and adversarial images as inputs, we propose a Two-stream Restoration Network (TRN) to restore the adversarial images. To optimally restore …
Robustness is an increasingly important property of machine learning models as they become more and more prevalent. We propose a defense against adversarial examples based on a k-nearest neighbor (kNN) on the intermediate activation of neural networks. Our scheme surpasses state-of-the-art defenses on MNIST and CIFAR-10 against l2-perturbation by a significant margin. With our models, the mean perturbation norm required to fool our MNIST model is 3.07 and 2.30 on CIFAR-10. Additionally, we propose a simple certifiable lower bound on the l2-norm of the adversarial perturbation using a more spec…
IBM Research（IBM研究）；Bar Ilan University（巴伊兰大学）；Western Digital（西部数据）
The results of a learning process depend on the input data. There are cases in which an adversary can strategically tamper with the input data to affect the outcome of the learning process. While some datasets are difficult to attack, many others are susceptible to manipulation. A resourceful attacker can tamper with large portions of the dataset and affect them. An attacker can additionally strategically focus on a preferred subset of the attributes in the dataset to maximize the effectiveness of the attack and minimize the resources allocated to data manipulation. In light of this vulnerabil…
DualGuard: Dual-stream Large Language Model Watermarking Defense against Paraphrase and Spoofing Attack
Institute of Information Engineering, Chinese Academy of Sciences（信息工程研究所，中国科学院）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络与信息安全学院）；Institute of Computing Technology, Chinese Academy of Sciences（计算技术研究所，中国科学院）
We propose a simple change to existing neural network structures for better defending against gradient-based adversarial attacks. Instead of using popular activation functions (such as ReLU), we advocate the use of k-Winners-Take-All (k-WTA) activation, a C0 discontinuous function that purposely invalidates the neural network model's gradient at densely distributed input data points. The proposed k-WTA activation can be readily used in nearly all existing networks and training methods with no significant overhead. Our proposal is theoretically rationalized. We analyze why the discontinuities i…
Defending the Edge: Representative-Attention Defense against Backdoor Attacks in Federated Learning
Dept. of Computer Science Faculty of Engineering University of Manchester Manchester, United Kingdom；Dept. of Computer Science Faculty of Engineering University of Manchester \& MBZUAI Abu Dhabi, United Arab Emirates；Dept. of Computer Science Faculty of Engineering University of Manchester \& KU Leuven Manchester, United Kingdom
School of Cyber Science and Engineering, Shanghai Jiao Tong University（上海交通大学网络科学与工程学院）；HUAWEI International, Singapore（华为国际（新加坡））
Deep learning has shown impressive performance on challenging perceptual tasks and has been widely used in software to provide intelligent services. However, researchers found deep neural networks vulnerable to adversarial examples. Since then, many methods are proposed to defend against adversaries in inputs, but they are either attack-dependent or shown to be ineffective with new attacks. And most of existing techniques have complicated structures or mechanisms that cause prohibitively high overhead or latency, impractical to apply on real software. We propose DAFAR, a feedback framework tha…
Your Model Is Leaking: Covert Information Transfer through LLM Residual Streams
University of Turku（图尔库大学）；City University of Macau（澳门城市大学）；University of Technology Sydney（悉尼科技大学）；CSIRO（澳大利亚联邦科学与工业研究组织）；Edith Cowan University（伊迪丝·考恩大学）
