Many-shot jailbreaking (MSJ) is an adversarial technique that exploits the long context windows of modern LLMs to circumvent model safety training by including in the prompt many examples of a "fake" assistant responding inappropriately before the final request. With enough examples, the model's in-context learning abilities override its safety training, and it responds as if it were the "fake" assistant. In this work, we probe the effectiveness of different fine-tuning and input sanitization approaches on mitigating MSJ attacks, alone and in combination. We find incremental mitigation effectiveness for each, and show that the combined techniques significantly reduce the effectiveness of MSJ attacks, while retaining model performance in benign in-context learning and conversational tasks. We suggest that our approach could meaningfully ameliorate this vulnerability if incorporated into model safety post-training.
Mitigating Many-shot Jailbreak Attacks with One Single Demonstration
Zhejiang University(浙江大学);Nanyang Technological University(南洋理工大学);Sun Yat-sen University(中山大学);University of Science and Technology of China(中国科学技术大学);Virginia Tech(弗吉尼亚理工大学)
PANDAS: Improving Many-shot Jailbreaking via Positive Affirmation, Negative Demonstration, and Adaptive Sampling
University of Toronto, Vector Institute(多伦多大学,向量研究所);University of Oxford(牛津大学);Polytechnique Montréal, Mila - Quebec AI Institute, University of Toronto(蒙特利尔理工学院,魁北克人工智能研究所,多伦多大学)
Towards Mitigating Systematics in Large-Scale Surveys via Few-Shot Optimal Transport-Based Feature Alignment
Space Telescope Science Institute(太空望远镜科学研究所);South African Radio Astronomy Observatory(南非射电天文台);University of the Western Cape(南非西部大学);Johns Hopkins University(约翰霍普金斯大学)
More is not always better? Enhancing Many-Shot In-Context Learning with Differentiated and Reweighting Objectives
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学人工智能学院);MoonshotAI;Xiaomi AI Lab(小米AI实验室);University of Electronic Science and Technology of China(电子科技大学);Mohamed bin Zayed University of Artificial Intelligence(Mohamed bin Zayed人工智能大学);School of Artifcial Intelligence, Wuhan University(武汉大学人工智能学院);Engineering Research Center of Next-Generation Intelligent Search and Recommendation, MoE(下一代智能搜索与推荐工程技术研究中心,MoE)