What Makes and Breaks Safety Fine-tuning? A Mechanistic Study
Five AI Ltd.(Five AI有限公司);University of Michigan(密歇根大学);Harvard University(哈佛大学);University of Oxford(牛津大学);Max Planck Institute for Intelligent Systems(马克斯·普朗克智能系统研究所);University of Copenhagen(哥本哈根大学)
Samyak Jain, Ekdeep Singh Lubana, Kemal Oksuz, Tom Joy, Philip H. S. Torr, Amartya Sanyal, Puneet K. Dokania
Safety fine-tuning helps align Large Language Models (LLMs) with human preferences for their safe deployment. To better understand the underlying factors that make models safe via safety fine-tuning, we design a synthetic data generation framework that captures salient aspects of an unsafe input by modeling the interaction between the task the model is asked to perform (e.g., "design") versus the specific concepts the task is asked to be performed upon (e.g., a "cycle" vs. a "bomb"). Using this, we investigate three well-known safety fine-tuning methods -- supervised safety fine-tuning, direct preference optimization, and unlearning -- and provide significant evidence demonstrating that these methods minimally transform MLP weights to specifically align unsafe inputs into its weights' null space. This yields a clustering of inputs based on whether the model deems them safe or not. Correspondingly, when an adversarial input (e.g., a jailbreak) is provided, its activations are closer to safer samples, leading to the model processing such an input as if it were safe. We validate our findings, wherever possible, on real-world models -- specifically, Llama-2 7B and Llama-3 8B.
Towards Understanding Safety Alignment: A Mechanistic Perspective from Safety Neurons
Department of Computer Science and Technology, BNRist(计算机科学与技术系,BNRist);Shenzhen International Graduate School(深圳国际研究生院);KIRC, Institute for Artificial Intelligence, Tsinghua University(人工智能研究院,清华大学)
Behind Harmful Compliance: Behavioral and Mechanistic Divergence Across LLM Jailbreaks
Department of Computer Science(计算机科学系);Luddy School of Informatics, Computing, and Engineering(信息学、计算与工程学院);Indiana University Bloomington(印第安纳大学布卢明顿分校)
How Emotion Shapes the Behavior of LLMs and Agents: A Mechanistic Study
School of Computer Science and Engineering, Beihang University(北京航空航天大学计算机科学与工程学院);College of Computing and Data Science, Nanyang Technological University(南洋理工大学计算与数据科学学院)
The Reasoning Trap -- Logical Reasoning as a Mechanistic Pathway to Situational Awareness
MARS 4.0 Fellowship, Cambridge AI Safety Hub(CAISH), University of Cambridge(MARS 4.0 Fellow,剑桥人工智能安全中心(CAISH),剑桥大学);AWS Generative AI Innovation Center, Amazon Web Services, USA(亚马逊生成AI创新中心,亚马逊网络服务,美国);Google, USA(谷歌,美国);Stanford University(斯坦福大学);Northeastern University, Seattle, WA, USA(东北大学,西雅图,华盛顿州,美国)
Inducing language models to assert their own consciousness restores human beliefs and values
Google(谷歌);University of Chicago(芝加哥大学);University of London(伦敦大学);University of Washington(华盛顿大学);Northwestern University(西北大学);Santa Fe Institute(圣达菲研究所)
Theory of Mind and Self-Attributions of Mentality are Dissociable in LLMs
Google, Paradigms of Intelligence Team(Google,智能范式团队);Knowledge Lab, University of Chicago(芝加哥大学知识实验室);Institute of Philosophy, School of Advanced Study, University of London(伦敦大学高等研究院哲学研究所);Department of Biomedical Informatics and Medical Education and Department of Bioethics and Humanities, School of Medicine, University of Washington(华盛顿大学医学院生物医学信息学与医学教育系及生物伦理学与人文学系);Kellogg School of Management, Northwestern University(西北大学凯洛格管理学院);Santa Fe Institute(圣塔菲研究所)
Mechanistic Interpretability of Fine-Tuned Vision Transformers on Distorted Images: Decoding Attention Head Behavior for Transparent and Trustworthy AI
Krembil Research Institute, University Health Network(肯贝尔研究所,大学健康网络)
Safe-SAIL: Towards a Fine-grained Safety Landscape of Large Language Models via Sparse Autoencoder Interpretation Framework
Alibaba Group(阿里巴巴集团);The State Key Laboratory of Blockchain and Data Security, Zhejiang University(浙江大学区块链与数据安全国家重点实验室);Language Technology Lab, University of Cambridge(剑桥大学语言技术实验室);Renmin University of China(中国人民大学)
Mechanistic Interpretability of Emotion Inference in Large Language Models
Institute for Creative Technologies, University of Southern California (USC)(创意技术研究所,南加州大学);Information Sciences Institute, University of Southern California (USC)(信息科学研究所,南加州大学);Department of Computer Science, University of Southern California (USC)(计算机科学系,南加州大学);Department of Statistics and Data Science, University of California, Los Angeles (UCLA)(统计与数据科学系,加州大学洛杉矶分校)
Mitigating LLM Over-Refusal via Dynamic Semantic Routing Calibration
School of Artificial Intelligence, Jilin University(吉林大学人工智能学院);CSIRO(澳大利亚联邦科学与工业研究组织);Center of Excellence for Generative AI, King Abdullah University of Science and Technology(阿卜杜拉国王科技大学生成式人工智能卓越中心)
Omni-Safety under Cross-Modality Conflict: Vulnerabilities, Dynamics Mechanisms and Efficient Alignment
Nanyang Technological University(南洋理工大学);Beijing University of Posts and Telecommunications(北京邮电大学);Tsinghua University(清华大学);Fudan University(复旦大学);University of Science and Technology of China(中国科学技术大学);Renmin University of China(中国人民大学)
When the Coffee Feature Activates on Coffins: An Analysis of Feature Extraction and Steering for Mechanistic Interpretability
Department of Philosophy of Nature and Technology(自然哲学与技术系);Munich School of Philosophy(慕尼黑哲学学院);Division of the Humanities and Social Sciences(人文与社会科学系);California Institute of Technology(加州理工学院)