Recently, Anil et al. (2024) show that many-shot (up to hundreds of) demonstrations can jailbreak state-of-the-art LLMs by exploiting their long-context capability. Nevertheless, is it possible to use few-shot demonstrations to efficiently jailbreak LLMs within limited context sizes? While the vanilla few-shot jailbreaking may be inefficient, we propose improved techniques such as injecting special system tokens like [/INST] and employing demo-level random search from a collected demo pool. These simple techniques result in surprisingly effective jailbreaking against aligned LLMs (even with advanced defenses). For examples, our method achieves >80% (mostly >95%) ASRs on Llama-2-7B and Llama-3-8B without multiple restarts, even if the models are enhanced by strong defenses such as perplexity detection and/or SmoothLLM, which is challenging for suffix-based jailbreaking. In addition, we conduct comprehensive and elaborate (e.g., making sure to use correct system prompts) evaluations against other aligned LLMs and advanced defenses, where our method consistently achieves nearly 100% ASRs. Our code is available at https://github.com/sail-sg/I-FSJ.
Scalable Defense against In-the-wild Jailbreaking Attacks with Safety Context Retrieval
School of EECS, Peking University（电子工程系，北京大学）；School of Mathematical Sciences, Peking University（数学系，北京大学）；State Key Lab of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University（通用人工智能国家重点实验室，智能科学与技术学校，北京大学）；Institute for Artificial Intelligence, Peking University（人工智能研究院，北京大学）
QueryAttack: Jailbreaking Aligned Large Language Models Using Structured Non-natural Query Language
Tsinghua Shenzhen International Graduate School（清华大学深圳国际研究生院）；Pengcheng Laboratory（鹏城实验室）；The Chinese University of Hong Kong（香港中文大学）；Jilin University（吉林大学）；Southwest University（西南大学）；University of Electronic Science and Technology of China（电子科技大学）；Shenzhen University（深圳大学）
Activation-Guided Local Editing for Jailbreaking Attacks
Beihang University（北航）；Hangzhou Innovation Institute（杭州创新研究院）；HKUST（香港科技大学）；South China University of Technology（华南理工大学）；Nanyang Technological University（南洋理工大学）；Kunming University of Science and Technology（昆明理工大学）
SelfDefend: LLMs Can Defend Themselves against Jailbreaking in a Practical Manner
The Hong Kong University of Science and Technology（香港科技大学）；University of Oregon（俄勒冈大学）；Nanyang Technological University（南洋理工大学）；City University of Hong Kong（香港城市大学）；HSBC（汇丰银行）
PANDAS: Improving Many-shot Jailbreaking via Positive Affirmation, Negative Demonstration, and Adaptive Sampling
University of Toronto, Vector Institute（多伦多大学，向量研究所）；University of Oxford（牛津大学）；Polytechnique Montréal, Mila - Quebec AI Institute, University of Toronto（蒙特利尔理工学院，魁北克人工智能研究所，多伦多大学）
Jailbreaking LLMs' Safeguard with Universal Magic Words for Text Embedding Models
Dept. of Comp. Sci. and Tech., Inst. for AI, Tsinghua-Bosch Joint ML Center, THBI Lab, BNRist Center, Tsinghua University, Beijing, 100084, China（计算机科学与技术系、人工智能研究所、清华-博世联合机器学习中心、THBI实验室、BNRist中心、清华大学、北京）；Department of Mathematical Sciences, Tsinghua University（数学科学系、清华大学）；BIMSA, Beijing, China（BIMSA、北京、中国）
Speechmatics（Speechmatics公司）；MATS（机器学习对齐理论与实验项目（注：MATS全称为Machine Learning Alignment Theory and Experiments，为独立研究机构，此处保留缩写并补充说明））；UCL（伦敦大学学院）；Stanford University（斯坦福大学）；University of Oxford（牛津大学）；Tangentic（Tangentic公司）；Anthropic（Anthropic公司）
Mind the Trojan Horse: Image Prompt Adapter Enabling Scalable and Deceptive Jailbreaking
School of Computer Science and Engineering, Sun Yat-Sen University（中山大学计算机科学与工程学院）；Nanyang Technological University（南洋理工大学）；Guangdong Province Key Laboratory of Information Security Technology（广东省信息安全技术重点实验室）
Jailbreaking Large Language Models Through Alignment Vulnerabilities in Out-of-Distribution Settings
University of Notre Dame（圣母大学）；Huazhong University of Science and Technology（华中科技大学）；Tsinghua University（清华大学）；Lehigh University（里海大学）；University of Illinois Chicago（芝加哥大学伊利诺伊分校）
Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study
Nanyang Technological University（南洋理工大学）；University of New South Wales（新南威尔士大学）；Virginia Tech（弗吉尼亚理工大学）；Huazhong University of Science and Technology（华中科技大学）
Feint and Attack: Attention-Based Strategies for Jailbreaking and Protecting LLMs
Beijing University of Posts and Telecommunications（北京邮电大学）；Hangzhou Dianzi University（杭州电子科技大学）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）；Fujian Cancer Hospital（福建癌症医院）
A Practical Framework for Evaluating Medical AI Security: Reproducible Assessment of Jailbreaking and Privacy Vulnerabilities Across Clinical Specialties
School of Computer Science and Technology, Guangdong University of Technology（广东工业大学计算机科学与技术学院）；School of Computer Science and Intelligence Education, Lingnan Normal University（岭南师范学院计算机与智能教育学院）；College of Computer, National University of Defense Technology（国防科技大学计算机学院）；School of Information Science and Technology, Guangdong University of Foreign Studies（广东外语外贸大学信息科学与技术学院）
Jailbreaking the Matrix: Nullspace Steering for Controlled Model Subversion
Department of Computer & Information Science & Engineering, University of Florida（佛罗里达大学计算机与信息科学与工程系）；School of Computer Science, University of Oklahoma（俄克拉荷马大学计算机科学学院）；Computer Science Laboratory, SRI International（SRI国际计算机科学实验室）
DualBreach: Efficient Dual-Jailbreaking via Target-Driven Initialization and Multi-Target Optimization
State Key Laboratory of Blockchain and Data Security, Zhejiang University, Hangzhou, China（区块链与数据安全国家重点实验室，浙江大学，杭州，中国）；Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security, Hangzhou, China（杭州高新技术区（滨江）区块链与数据安全研究院，杭州，中国）
PrisonBreak: Jailbreaking Large Language Models with at Most Twenty-Five Targeted Bit-flips
Oregon State University（俄勒冈州立大学）；University of British Columbia（不列颠哥伦比亚大学）；University of Toronto（多伦多大学）；George Mason University（乔治·梅森大学）；Rutgers University（罗格斯大学）；University of Texas at Arlington（德克萨斯大学阿灵顿分校）
OrchJail: Jailbreaking Tool-Calling Text-to-Image Agents by Orchestration-Guided Fuzzing
Institute of Software, Chinese Academy of Sciences, Beijing, China；Science \& Technology on Integrated Information System Laboratory, Beijing, China；State Key Laboratory of Complex System Modeling；University of Chinese Academy of Sciences, Beijing, China
