Large language models (LLMs) are expected to follow instructions from users and engage in conversations. Techniques to enhance LLMs' instruction-following capabilities typically fine-tune them using data structured according to a predefined chat template. Although chat templates are shown to be effective in optimizing LLM performance, their impact on safety alignment of LLMs has been less understood, which is crucial for deploying LLMs safely at scale.
In this paper, we investigate how chat templates affect safety alignment of LLMs. We identify a common vulnerability, named ChatBug, that is introduced by chat templates. Our key insight to identify ChatBug is that the chat templates provide a rigid format that need to be followed by LLMs, but not by users. Hence, a malicious user may not necessarily follow the chat template when prompting LLMs. Instead, malicious users could leverage their knowledge of the chat template and accordingly craft their prompts to bypass safety alignments of LLMs. We develop two attacks to exploit the ChatBug vulnerability. We demonstrate that a malicious user can exploit the ChatBug vulnerability of eight state-of-the-art (SOTA) LLMs and effectively elicit unintended responses from these models. Moreover, we show that ChatBug can be exploited by existing jailbreak attacks to enhance their attack success rates. We investigate potential countermeasures to ChatBug. Our results show that while adversarial training effectively mitigates the ChatBug vulnerability, the victim model incurs significant performance degradation. These results highlight the trade-off between safety alignment and helpfulness. Developing new methods for instruction tuning to balance this trade-off is an open and critical direction for future research
Exploring the Limits of ChatGPT in Software Security Applications
University of Michigan, Ann Arbor（密歇根大学安娜堡分校）；ASU（亚利桑那州立大学）；Washington University in St. Louis（圣路易斯华盛顿大学）；University of Wisconsin, Madison（威斯康星大学麦迪逊分校）
Can LLMs Help Allocate Public Health Resources? A Case Study on Childhood Lead Testing
Department of Computer and Information Sciences, Fordham University（福特汉姆大学计算机与信息科学系）；Department of Systems Engineering, Stevens Institute of Technology（史蒂文斯理工学院系统工程系）
Evaluating Large Language Models for Line-Level Vulnerability Localization
School of Software, Beihang University（北京航空航天大学软件学院）；Nanyang Technological University（南洋理工大学）；Yale University（耶鲁大学）；Singapore Management University（新加坡管理大学）
MixAT: Combining Continuous and Discrete Adversarial Training for LLMs
INSAIT, Sofia University "St. Kliment Ohridski"（INSAIT，索菲亚大学"圣克莱孟·奥赫里迪斯基"）；ETH Zurich（苏黎世联邦理工学院）；ELTE Eötvös Loránd University, Budapest, Hungary（ELTE 爱斯特霍特·洛兰德大学，布达佩斯，匈牙利）
Emerging Security Challenges of Large Language Models
Télécom SudParis（巴黎高等电信学院）；City University of New York（纽约城市大学）；Universität Liechtenstein（列支敦士登大学）；Imperial College London（伦敦帝国学院）；Universität der Bundeswehr München（慕尼黑联邦国防军大学）
Large Language Models (LLMs) and Generative AI in Cybersecurity and Privacy: A Survey of Dual-Use Risks, AI-Generated Malware, Explainability, and Defensive Strategies
Virelya Intelligence Research Labs（Virelya智能研究实验室）；Google（谷歌）
Encyclo-K: Evaluating LLMs with Dynamically Composed Knowledge Statements
School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Nanjing University（南京大学）；The University of Manchester（曼彻斯特大学）
Steering LLM Viewpoints through Fabricated Evidence Injection
The Hong Kong University of Science and Technology（香港科技大学）；University of Science and Technology of China（中国科学技术大学）；University of Liverpool（利物浦大学）；Guangzhou University（广州大学）
