Self and Cross-Model Distillation for LLMs: Effective Methods for Refusal Pattern Alignment
Jie Li, Yi Liu, Chongyang Liu, Xiaoning Ren, Ling Shi, Weisong Sun, Yinxing Xue
备注:
The method used in the paper has obvious problems and ambiguities. The security enhancement method we used cannot be considered distillation, but it is described as distillation in the paper, and the experiment lacks comparison and baseline, which has been criticized by many peers. In order to avoid further dissemination, we have decided to withdraw the paper
Large Language Models (LLMs) like OpenAI's GPT series, Anthropic's Claude, and Meta's LLaMa have shown remarkable capabilities in text generation. However, their susceptibility to toxic prompts presents significant security challenges. This paper investigates alignment techniques, including Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF), to mitigate these risks. We conduct an empirical study on refusal patterns across nine LLMs, revealing that models with uniform refusal patterns, such as Claude3, exhibit higher security. Based on these findings, we propose self-distilling and cross-model distilling methods to enhance LLM security. Our results show that these methods significantly improve refusal rates and reduce unsafe content, with cross-model distilling achieving refusal rates close to Claude3's 94.51%. These findings underscore the potential of distillation-based alignment in securing LLMs against toxic prompts.
Alignment Quality Index (AQI) : Beyond Refusals: AQI as an Intrinsic Alignment Diagnostic via Latent Geometry, Cluster Divergence, and Layer wise Pooled Representations
Manipal University Jaipur(Manipal大学贾ipur分校);LinkedIn;IIT Kharagpur(印度理工学院Kharagpur分校);IIIT Guwahati(印度冈瓦提理工学院);Texas A&M University(德克萨斯A&M大学);Vellore Institute of Technology(维洛雷理工学院);Meta AI;Evalueserve;New York University(纽约大学);Amazon AI;BITS Goa(果阿理工学院)
Omni-Safety under Cross-Modality Conflict: Vulnerabilities, Dynamics Mechanisms and Efficient Alignment
Nanyang Technological University(南洋理工大学);Beijing University of Posts and Telecommunications(北京邮电大学);Tsinghua University(清华大学);Fudan University(复旦大学);University of Science and Technology of China(中国科学技术大学);Renmin University of China(中国人民大学)
Decomposing Refusal Steering in Mixture-of-Experts Models
Department of Interdisciplinary Studies(交叉学科研究部);University of Montana(蒙大拿大学);Department of Pharmacy Practice & Science(药学与科学系);University of Arizona(亚利桑那大学);European Bioinformatics Institute(欧洲生物信息研究所);European Molecular Biology Laboratory(欧洲分子生物学实验室);Wellcome Genome Campus(沃氏基因组校园)
DualEdit: Mitigating Safety Fallback in LLM Backdoor Editing via Affirmation-Refusal Regulation
University of Science and Technology of China(中国科学技术大学);National University of Singapore(新加坡国立大学);Singapore Management University(新加坡管理大学);MoE Key Lab of BIPC, University of Science and Technology of China(中国科学技术大学信息与通信技术国家重点实验室)
CoT2Align: Cross-Chain of Thought Distillation via Optimal Transport Alignment for Language Models with Different Tokenizers
Hanoi University of Science and Technology(河内理工大学);Oraichain Labs Inc.(Oraichain实验室公司);ByteDance Inc(字节跳动公司);VNU University of Engineering and Technology(越南国家大学工程技术大学);Monash University(莫纳什大学);University of Oregon(俄勒冈大学)
Does Reasoning Preserve Alignment? On the Trustworthiness of Large Reasoning Models
University of Colorado Boulder(科罗拉多大学博尔德分校);University of Central Florida(中佛罗里达大学);University of Maryland College Park(马里兰大学帕克分校);University of Wisconsin-Madison(威斯康星大学麦迪逊分校)
Enhancing Knowledge Distillation of Large Language Models through Efficient Multi-Modal Distribution Alignment
School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Wuhan AI Research(武汉人工智能研究院);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
Feature Alignment-Based Knowledge Distillation for Efficient Compression of Large Language Models
Purdue University(普渡大学);New York University(纽约大学);Stevens Institute of Technology(史蒂文斯理工学院);Northeastern University(东北大学);The Chinese University of Hong Kong(香港中文大学)
TALAS: Teacher-Anchored Layer Alignment with Adaptive Sharpness-Aware Minimization for Embedding Distillation
Hanoi University of Science and Technology(河内科技大学);VNU University of Engineering and Technology(越南国立大学工程与技术大学);University of Oregon(俄勒冈大学);Monash University(莫纳什大学)
Long-Chain Reasoning Distillation via Adaptive Prefix Alignment
School of Computer Science and Engineering, Northeastern University, China(东北大学计算机科学与工程学院);Department of Computer Science and Technology, Institute for AI, Tsinghua University, China(清华大学计算机科学与技术系);Alibaba Group, China(阿里巴巴集团)
AdvChain: Adversarial Chain-of-Thought Tuning for Robust Safety Alignment of Large Reasoning Models
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));State University of New York at Buffalo(纽约州立大学布法罗分校);Huawei International, Singapore(新加坡华为国际)
Beyond Surface Alignment: Rebuilding LLMs Safety Mechanism via Probabilistically Ablating Refusal Direction
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络安全学院)