Large language models (LLMs) have seen rapid development in recent years, revolutionizing various applications and significantly enhancing convenience and productivity. However, alongside their impressive capabilities, ethical concerns and new types of attacks, such as jailbreaking, have emerged. While most prompting techniques focus on optimizing adversarial inputs for individual cases, resulting in higher computational costs when dealing with large datasets. Less research has addressed the more general setting of training a universal attacker that can transfer to unseen tasks. In this paper, we introduce JUMP, a prompt-based method designed to jailbreak LLMs using universal multi-prompts. We also adapt our approach for defense, which we term DUMP. Experimental results demonstrate that our method for optimizing universal multi-prompts outperforms existing techniques.
Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study
Nanyang Technological University(南洋理工大学);University of New South Wales(新南威尔士大学);Virginia Tech(弗吉尼亚理工大学);Huazhong University of Science and Technology(华中科技大学)
Speechmatics(Speechmatics公司);MATS(机器学习对齐理论与实验项目(注:MATS全称为Machine Learning Alignment Theory and Experiments,为独立研究机构,此处保留缩写并补充说明));UCL(伦敦大学学院);Stanford University(斯坦福大学);University of Oxford(牛津大学);Tangentic(Tangentic公司);Anthropic(Anthropic公司)
PANDAS: Improving Many-shot Jailbreaking via Positive Affirmation, Negative Demonstration, and Adaptive Sampling
University of Toronto, Vector Institute(多伦多大学,向量研究所);University of Oxford(牛津大学);Polytechnique Montréal, Mila - Quebec AI Institute, University of Toronto(蒙特利尔理工学院,魁北克人工智能研究所,多伦多大学)
Scalable Defense against In-the-wild Jailbreaking Attacks with Safety Context Retrieval
School of EECS, Peking University(电子工程系,北京大学);School of Mathematical Sciences, Peking University(数学系,北京大学);State Key Lab of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University(通用人工智能国家重点实验室,智能科学与技术学校,北京大学);Institute for Artificial Intelligence, Peking University(人工智能研究院,北京大学)
The State Key Laboratory of Blockchain and Data Security, Zhejiang University(区块链与数据安全国家重点实验室,浙江大学);Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security(杭州高新技术区(滨江)区块链与数据安全研究院);Nanyang Technological University(南洋理工大学);King Abdullah University of Science and Technology(国王阿卜杜勒阿齐兹科技大学);SUN YAT-SEN UNIVERSITY(孙中山大学)
Jailbreaking Large Language Models Through Alignment Vulnerabilities in Out-of-Distribution Settings
University of Notre Dame(圣母大学);Huazhong University of Science and Technology(华中科技大学);Tsinghua University(清华大学);Lehigh University(里海大学);University of Illinois Chicago(芝加哥大学伊利诺伊分校)
Mind the Trojan Horse: Image Prompt Adapter Enabling Scalable and Deceptive Jailbreaking
School of Computer Science and Engineering, Sun Yat-Sen University(中山大学计算机科学与工程学院);Nanyang Technological University(南洋理工大学);Guangdong Province Key Laboratory of Information Security Technology(广东省信息安全技术重点实验室)
OrchJail: Jailbreaking Tool-Calling Text-to-Image Agents by Orchestration-Guided Fuzzing
Institute of Software, Chinese Academy of Sciences, Beijing, China;Science \& Technology on Integrated Information System Laboratory, Beijing, China;State Key Laboratory of Complex System Modeling;University of Chinese Academy of Sciences, Beijing, China
Jailbreaking LLMs' Safeguard with Universal Magic Words for Text Embedding Models
Dept. of Comp. Sci. and Tech., Inst. for AI, Tsinghua-Bosch Joint ML Center, THBI Lab, BNRist Center, Tsinghua University, Beijing, 100084, China(计算机科学与技术系、人工智能研究所、清华-博世联合机器学习中心、THBI实验室、BNRist中心、清华大学、北京);Department of Mathematical Sciences, Tsinghua University(数学科学系、清华大学);BIMSA, Beijing, China(BIMSA、北京、中国)
Continuous Embedding Attacks via Clipped Inputs in Jailbreaking Large Language Models
University of New South Wales(新南威尔士大学);Delft University of Technology(代尔夫特理工大学);Nanyang Technological University(南洋理工大学);Huazhong University of Science and Technology(华中科技大学);Radboud University(拉德堡德大学)