State-of-the-art language model fine-tuning techniques, such as Direct Preference Optimization (DPO), restrict user control by hard-coding predefined behaviors into the model. To address this, we propose a novel method, Configurable Safety Tuning (CST), that augments DPO using synthetic preference data to facilitate flexible safety configuration of LLMs at inference time. CST overcomes the constraints of vanilla DPO by introducing a system prompt specifying safety configurations, enabling LLM deployers to disable/enable safety preferences based on their need, just changing the system prompt. Our experimental evaluations indicate that CST successfully manages different safety configurations and retains the original functionality of LLMs, showing it is a robust method for configurable deployment. Data and models available at https://github.com/vicgalle/configurable-safety-tuning
MidPO: Dual Preference Optimization for Safety and Helpfulness in Large Language Models via a Mixture of Experts Framework
Sun Yat-sen University(中山大学);Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(深圳先进技术研究院、中国科学院);Beijing Normal University(北京师范大学);Central University of Finance and Economics(中央财经大学)
CHILLGuard: Towards Fine-Grained Chinese LLM Safety Guardrail with Scalable Data Construction and Model-aware Preference Alignment
Tsinghua University(清华大学);Beijing Normal University(北京师范大学);South China University of Technology(华南理工大学);Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));Shenzhen ShenNong Information Technology Co., Ltd.(深圳神农信息技术有限公司)
More is Less: The Pitfalls of Multi-Model Synthetic Preference Data in DPO Safety Alignment
Purdue University(普渡大学);The University of Texas at Austin(德克萨斯大学奥斯汀分校);The University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校);University of California, Los Angeles(加州大学洛杉矶分校)
DOG-DPO:Dynamic Optimization in Geometry for Safety Alignment
University of Southern California(南加州大学);Iowa State University(爱荷华州立大学);University of Wisconsin–Madison(威斯康星大学麦迪逊分校);UT Austin(德克萨斯大学奥斯汀分校);Independent Researcher(独立研究员);University of Notre Dame(圣母大学)
The Rise of Darkness: Safety-Utility Trade-Offs in Role-Playing Dialogue Agents
Institute of Computing and Intelligence, Harbin Institute of Technology(哈尔滨工业大学计算与智能学院);Baidu Inc.(百度公司);College of Intelligence and Computing, Tianjin University(天津大学智能与计算学院)
Multi-Level Safety Continual Projection for Fine-Tuned Large Language Models without Retraining
Beijing Key Laboratory of Safe AI and Superalignment(北京安全人工智能与超对齐重点实验室);Beijing Institute of AI Safety and Governance(北京人工智能安全与治理研究院);Brain-inspired Cognitive AI Lab, Institute of Automation, Chinese Academy of Sciences(脑启发认知人工智能实验室,中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);Long-term AI(长期人工智能)
TherapyGym: Evaluating and Aligning Clinical Fidelity and Safety in Therapy Chatbots
Department of Computer Science, Stanford University, Stanford, CA, USA(计算机科学系,斯坦福大学,斯坦福,加州,美国);Department of Psychiatry and Behavioral Sciences, Stanford University, Stanford, CA, USA(精神病学与行为科学系,斯坦福大学,斯坦福,加州,美国);Department of Biomedical Data Science, Stanford University, Stanford, CA, USA(生物医学数据科学系,斯坦福大学,斯坦福,加州,美国);University of California, Berkeley, Berkeley, CA, USA(加州大学伯克利分校,伯克利,加州,美国);The University of Hong Kong, Hong Kong(香港大学,香港)
The Art of (Mis)alignment: How Fine-Tuning Methods Effectively Misalign and Realign LLMs in Post-Training
University of Electronic Science and Technology of China(电子科技大学);Flexera;CISPA Helmholtz Center for Information Security(CISPA亥姆霍兹信息安全中心);Nanyang Technological University(南洋理工大学)
Multi-Objective Alignment of Language Models for Personalized Psychotherapy
Department of Computer Science, University of California, Los Angeles, CA, USA(计算机科学系,加州大学洛杉矶分校);Department of Psychology, The University of Texas at Austin, Austin, TX, USA(心理学系,德克萨斯大学奥斯汀分校);Department of Psychiatry, NYU Grossman School of Medicine, New York, NY, USA(精神病学系,纽约大学格罗斯曼医学院)