Public LLMs such as the Llama 2-Chat underwent alignment training and were considered safe. Recently Qi et al. [2024] reported that even benign fine-tuning on seemingly safe datasets can give rise to unsafe behaviors in the models. The current paper is about methods and best practices to mitigate such loss of alignment. We focus on the setting where a public model is fine-tuned before serving users for specific usage, where the model should improve on the downstream task while maintaining alignment. Through extensive experiments on several chat models (Meta's Llama 2-Chat, Mistral AI's Mistral 7B Instruct v0.2, and OpenAI's GPT-3.5 Turbo), this paper uncovers that the prompt templates used during fine-tuning and inference play a crucial role in preserving safety alignment, and proposes the ``Pure Tuning, Safe Testing'' (PTST) strategy -- fine-tune models without a safety prompt, but include it at test time. This seemingly counterintuitive strategy incorporates an intended distribution shift to encourage alignment preservation. Fine-tuning experiments on GSM8K, ChatDoctor, and OpenOrca show that PTST significantly reduces the rise of unsafe behaviors.
The Art of Socratic Inquiry: A Framework for Proactive Template-Guided Therapeutic Conversation Generation
School of Information Science and Engineering, Lanzhou University（信息科学与工程学院，兰州大学）；School of Medical Technology, Beijing Institute of Technology（医学技术学院，北京理工大学）
Voices Across Registers: Corpus-Conditioned Vernacular Jailbreaks against Aligned LLMs via Fanfiction Subgenres
School of Science and Engineering, The Chinese University of Hong Kong (Shenzhen)（香港中文大学（深圳）科学与工程学院）；The Shenzhen Future Network of Intelligence Institute (FNii-Shenzhen)（深圳未来网络智能研究院）；The Guangdong Provincial Key Laboratory of Future Networks of Intelligence（广东省未来网络智能重点实验室）；School of Microelectronics, Xi’an Jiaotong University（西安交通大学微电子学院）
SeqAR: Jailbreak LLMs with Sequential Auto-Generated Characters
Shanghai University of Finance and Economics（上海财经大学）；Southern University of Science and Technology（南方科技大学）；Ant Group（蚂蚁集团）；The Chinese University of Hong Kong（香港中文大学）
ConnectED: A Curriculum-Aligned AI System for Vietnamese Instructional Lesson Planning and Student Learning
Secomus Technology（赛科默斯科技）；FPT Software AI Center（FPT软件人工智能中心）；Viettel Solutions（越南电信解决方案公司）；Vietnam Ernst & Young Limited Company（越南安永有限公司）；Banking Academy of Vietnam（越南银行学院）；Vietnam Academy of Science and Technology（越南科学技术研究院）；University of Koblenz（科布伦茨大学）
