We demonstrate that preference optimization methods can effectively enhance LLM safety. Applying various alignment techniques to the Falcon 11B model using safety datasets, we achieve a significant boost in global safety score (from $57.64\%$ to $99.90\%$) as measured by LlamaGuard 3 8B, competing with state-of-the-art models. On toxicity benchmarks, average scores in adversarial settings dropped from over $0.6$ to less than $0.07$. However, this safety improvement comes at the cost of reduced general capabilities, particularly in math, suggesting a trade-off. We identify noise contrastive alignment (Safe-NCA) as an optimal method for balancing safety and performance. Our study ultimately shows that alignment techniques can be sufficient for building safe and robust models.
Towards Understanding Safety Alignment: A Mechanistic Perspective from Safety Neurons
Department of Computer Science and Technology, BNRist（计算机科学与技术系，BNRist）；Shenzhen International Graduate School（深圳国际研究生院）；KIRC, Institute for Artificial Intelligence, Tsinghua University（人工智能研究院，清华大学）
Regularized Conditional Diffusion Model for Multi-Task Preference Alignment
Harbin Institute of Technology（哈尔滨工业大学）；Institute of Artificial Intelligence (TeleAI), China Telecom（中国电信人工智能研究院（TeleAI））；Hong Kong University of Science and Technology（香港科技大学）
Data Selection for LLM Alignment Using Fine-Grained Preferences
National Key Laboratory for Novel Software Technology, Nanjing University（南京大学新型软件技术国家重点实验室）；School of Artificial Intelligence, Nanjing University（南京大学人工智能学院）；Algorithm Tech, Taobao & Tmall Group of Alibaba（阿里巴巴集团算法技术部）；School of Intelligence Science and Technology, Nanjing University（南京大学智能科学与技术学院）
SPO: Multi-Dimensional Preference Sequential Alignment With Implicit Reward Modeling
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Baichuan Inc.（百川智能）
Safety Alignment Depth in Large Language Models: A Markov Chain Perspective
Institute of Information Science (IIS), Academia Sinica（中央研究院资讯科学研究所）；National Taiwan University（国立台湾大学）；National Yang Ming Chiao Tung University（国立阳明交通大学）
PEARL: Zero-shot Cross-task Preference Alignment and Robust Reward Learning for Robotic Manipulation
Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；King’s College London（伦敦国王学院）；Beijing Institute for General AI（北京通用人工智能研究院）；Shanghai Jiao Tong University（上海交通大学）
Preference Alignment on Diffusion Model: A Comprehensive Survey for Image Generation and Editing
University of Liverpool（利物浦大学）；Institute of Software Chinese Academy of Sciences（中国科学院软件研究所）；University of Edinburgh（爱丁堡大学）；University of Bristol（布里斯托大学）；University of Exeter（埃克塞特大学）
PREFINE: Preference-Based Implicit Reward and Cost Fine-Tuning for Safety Alignment
TCS Research, \ of CSE, IIT Madras India；Department of Computing Science, \ of Alberta Canada；Qatar Computing Research Institute, \ Bin Khalifa University Qatar；Department of Data Science \& AI, Wadhwani School of Data Science \& AI, IIT Madras India；TCS Research, \ of CSE, IIT Madras；Department of Computing Science, \ of Alberta；Qatar Computing Research Institute, \ Bin Khalifa University；Department of Data Science \& AI, Wadhwani School of Data Science \& AI, IIT Madras
DOG-DPO:Dynamic Optimization in Geometry for Safety Alignment
University of Southern California（南加州大学）；Iowa State University（爱荷华州立大学）；University of Wisconsin–Madison（威斯康星大学麦迪逊分校）；UT Austin（德克萨斯大学奥斯汀分校）；Independent Researcher（独立研究员）；University of Notre Dame（圣母大学）
Harvard University（哈佛大学）；University of California Berkeley（加州大学伯克利分校）；Massachusetts Institute of Technology（麻省理工学院）；Ikigai Labs（Ikigai实验室）；University of California San Diego（加州大学圣地亚哥分校）
Differentially Private Preference Data Synthesis for Large Language Model Alignment
Department of Computer Science, University of Virginia, Charlottesville, Virginia, USA（弗吉尼亚大学计算机科学系）；Department of Electrical and Computer Engineering, University of Virginia, Charlottesville, Virginia, USA（弗吉尼亚大学电气与计算机工程系）
Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment
Artificial Intelligence Research Laboratory, Pennsylvania State University（宾夕法尼亚州立大学人工智能研究实验室）；University of Chinese Academy of Sciences（中国科学院大学）；Tencent AI Lab（腾讯人工智能实验室）
