The risk of harmful content generated by large language models (LLMs) becomes a critical concern. This paper presents a systematic study on assessing and improving LLMs' capability to perform the task of \textbf{course-correction}, \ie, the model can steer away from generating harmful content autonomously. To start with, we introduce the \textsc{C$^2$-Eval} benchmark for quantitative assessment and analyze 10 popular LLMs, revealing varying proficiency of current safety-tuned LLMs in course-correction. To improve, we propose fine-tuning LLMs with preference learning, emphasizing the preference for timely course-correction. Using an automated pipeline, we create \textsc{C$^2$-Syn}, a synthetic dataset with 750K pairwise preferences, to teach models the concept of timely course-correction through data-driven preference learning. Experiments on 2 LLMs, \textsc{Llama2-Chat 7B} and \textsc{Qwen2 7B}, show that our method effectively enhances course-correction skills without affecting general performance. Additionally, it effectively improves LLMs' safety, particularly in resisting jailbreak attacks.
DOG-DPO:Dynamic Optimization in Geometry for Safety Alignment
University of Southern California（南加州大学）；Iowa State University（爱荷华州立大学）；University of Wisconsin–Madison（威斯康星大学麦迪逊分校）；UT Austin（德克萨斯大学奥斯汀分校）；Independent Researcher（独立研究员）；University of Notre Dame（圣母大学）
MidPO: Dual Preference Optimization for Safety and Helpfulness in Large Language Models via a Mixture of Experts Framework
Sun Yat-sen University（中山大学）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（深圳先进技术研究院、中国科学院）；Beijing Normal University（北京师范大学）；Central University of Finance and Economics（中央财经大学）
CHILLGuard: Towards Fine-Grained Chinese LLM Safety Guardrail with Scalable Data Construction and Model-aware Preference Alignment
Tsinghua University（清华大学）；Beijing Normal University（北京师范大学）；South China University of Technology（华南理工大学）；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；Shenzhen ShenNong Information Technology Co., Ltd.（深圳神农信息技术有限公司）
More is Less: The Pitfalls of Multi-Model Synthetic Preference Data in DPO Safety Alignment
Purdue University（普渡大学）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）；The University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；University of California, Los Angeles（加州大学洛杉矶分校）
Multi-Level Safety Continual Projection for Fine-Tuned Large Language Models without Retraining
Beijing Key Laboratory of Safe AI and Superalignment（北京安全人工智能与超对齐重点实验室）；Beijing Institute of AI Safety and Governance（北京人工智能安全与治理研究院）；Brain-inspired Cognitive AI Lab, Institute of Automation, Chinese Academy of Sciences（脑启发认知人工智能实验室，中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Long-term AI（长期人工智能）
Multi-Objective Alignment of Language Models for Personalized Psychotherapy
Department of Computer Science, University of California, Los Angeles, CA, USA（计算机科学系，加州大学洛杉矶分校）；Department of Psychology, The University of Texas at Austin, Austin, TX, USA（心理学系，德克萨斯大学奥斯汀分校）；Department of Psychiatry, NYU Grossman School of Medicine, New York, NY, USA（精神病学系，纽约大学格罗斯曼医学院）
What Makes and Breaks Safety Fine-tuning? A Mechanistic Study
Five AI Ltd.（Five AI有限公司）；University of Michigan（密歇根大学）；Harvard University（哈佛大学）；University of Oxford（牛津大学）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；University of Copenhagen（哥本哈根大学）
The Rise of Darkness: Safety-Utility Trade-Offs in Role-Playing Dialogue Agents
Institute of Computing and Intelligence, Harbin Institute of Technology（哈尔滨工业大学计算与智能学院）；Baidu Inc.（百度公司）；College of Intelligence and Computing, Tianjin University（天津大学智能与计算学院）
The Art of (Mis)alignment: How Fine-Tuning Methods Effectively Misalign and Realign LLMs in Post-Training
University of Electronic Science and Technology of China（电子科技大学）；Flexera；CISPA Helmholtz Center for Information Security（CISPA亥姆霍兹信息安全中心）；Nanyang Technological University（南洋理工大学）
