The safety alignment of current Large Language Models (LLMs) is vulnerable. Relatively simple attacks, or even benign fine-tuning, can jailbreak aligned models. We argue that many of these vulnerabilities are related to a shared underlying issue: safety alignment can take shortcuts, wherein the alignment adapts a model's generative distribution primarily over only its very first few output tokens. We refer to this issue as shallow safety alignment. In this paper, we present case studies to explain why shallow safety alignment can exist and provide evidence that current aligned LLMs are subject to this issue. We also show how these findings help explain multiple recently discovered vulnerabilities in LLMs, including the susceptibility to adversarial suffix attacks, prefilling attacks, decoding parameter attacks, and fine-tuning attacks. Importantly, we discuss how this consolidated notion of shallow safety alignment sheds light on promising research directions for mitigating these vulnerabilities. For instance, we show that deepening the safety alignment beyond just the first few tokens can often meaningfully improve robustness against some common exploits. Finally, we design a regularized finetuning objective that makes the safety alignment more persistent against fine-tuning attacks by constraining updates on initial tokens. Overall, we advocate that future safety alignment should be made more than just a few tokens deep.
Safety Alignment Should Be Made More Than Just A Few Attention Heads
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络安全学院）
Birkbeck, University of London（伦敦大学伯贝克学院）；School of Advanced Study, Institute of Philosophy, University of London（伦敦大学高等研究院哲学研究所）；University of Utah（犹他大学）；Tufts University（塔夫茨大学）
Beyond Token-Level Guidance: Inference-Time Alignment of Specialized LLMs via Cross-Family Representation Steering
College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算与数据科学学院）；College of Cryptology and Cyber Science, Nankai University（南开大学密码与网络科学学院）
The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMs
Beijing University of Posts and Telecommunications（北京邮电大学）；Jinan University（暨南大学）；Beihang University（北京航空航天大学）；China Academy of Information and Communications Technology（中国信息通信研究院）
SafeSteer: Localized On-Policy Distillation for Efficient Safety Alignment
Beihang University（北航）；Beijing Institute of Technology（北京理工大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；Peking University（北京大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）
TRACE: Trajectory Return Attribution and Contrastive Erasure for Multi-Turn Safety
King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）；University of Macau（澳门大学）；RIKEN Center for Advanced Intelligence Project（理化学研究所先进智能研究中心）；University of Electronic Science and Technology of China（电子科技大学）
Risk Awareness Injection: Calibrating Vision-Language Models for Safety without Compromising Utility
Shien-Ming Wu School of Intelligent Engineering, South China University of Technology（华南理工大学吴贤铭智能工程学院）；Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)（广东省人工智能与数字经济实验室（深圳））；Computer Science and Engineering, The Hong Kong University of Science and Technology（香港科技大学计算机科学与工程系）；University of Electronic Science and Technology of China（电子科技大学）
