The rise of "jailbreak" attacks on language models has led to a flurry of defenses aimed at preventing undesirable responses. We critically examine the two stages of the defense pipeline: (i) defining what constitutes unsafe outputs, and (ii) enforcing the definition via methods such as input processing or fine-tuning. To test the efficacy of existing enforcement mechanisms, we consider a simple and well-specified definition of unsafe outputs--outputs that contain the word "purple". Surprisingly, existing fine-tuning and input defenses fail on this simple problem, casting doubt on whether enforcement algorithms can be robust for more complicated definitions. We find that real safety benchmarks similarly test enforcement for a fixed definition. We hope that future research can lead to effective/fast enforcement as well as high quality definitions used for enforcement and evaluation.
PandaGuard: Systematic Evaluation of LLM Safety against Jailbreaking Attacks
Beijing Institute of AI Safety and Governance (Beijing-AISI)(北京人工智能安全与治理研究院);Beijing Key Laboratory of Safe AI and Superalignment(北京安全人工智能与超对齐重点实验室);BrainCog Lab, CASIA(CASIA脑认知实验室)
Activation-Guided Local Editing for Jailbreaking Attacks
Beihang University(北航);Hangzhou Innovation Institute(杭州创新研究院);HKUST(香港科技大学);South China University of Technology(华南理工大学);Nanyang Technological University(南洋理工大学);Kunming University of Science and Technology(昆明理工大学)
Scalable Defense against In-the-wild Jailbreaking Attacks with Safety Context Retrieval
School of EECS, Peking University(电子工程系,北京大学);School of Mathematical Sciences, Peking University(数学系,北京大学);State Key Lab of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University(通用人工智能国家重点实验室,智能科学与技术学校,北京大学);Institute for Artificial Intelligence, Peking University(人工智能研究院,北京大学)
Data to Defense: The Role of Curation in Customizing LLMs Against Jailbreaking Attacks
Stony Brook University(石溪大学);Binghamton University(宾汉姆顿大学);University of Iowa(爱荷华大学);Dartmouth College(达特茅斯学院);Georgia Institute of Technology(佐治亚理工学院)
Jailbreaking LLMs & VLMs: Mechanisms, Evaluation, and Unified Defense
School of Cyberspace Security, Beijing University of Posts and Telecommunications(北京邮电大学网络空间安全学院);China Academy of Information and Communications Technology(中国信息通信研究院)
TeleAI-Safety: A comprehensive LLM jailbreaking benchmark towards attacks, defenses, and evaluations
Institute of Artificial Intelligence (TeleAI) of China Telecom(中国电信人工智能研究院(TeleAI));Shanghai Jiao Tong University(上海交通大学);Sichuan University(四川大学);University of Chinese Academy of Sciences(中国科学院大学);Chinese Academy of Sciences(中国科学院);University of Science and Technology of China(中国科学技术大学);Peking University(北京大学);Harbin Institute of Technology(哈尔滨工业大学)
QueryAttack: Jailbreaking Aligned Large Language Models Using Structured Non-natural Query Language
Tsinghua Shenzhen International Graduate School(清华大学深圳国际研究生院);Pengcheng Laboratory(鹏城实验室);The Chinese University of Hong Kong(香港中文大学);Jilin University(吉林大学);Southwest University(西南大学);University of Electronic Science and Technology of China(电子科技大学);Shenzhen University(深圳大学)
DELMAN: Dynamic Defense Against Large Language Model Jailbreaking with Model Editing
ShanghaiTech University(上海科技大学);The State Key Laboratory of Blockchain and Data Security, Zhejiang University(浙江大学区块链与数据安全全国重点实验室);The CoAI group, Tsinghua University(清华大学CoAI课题组)
Jailbreaking and Mitigation of Vulnerabilities in Large Language Models
Hunan University Changsha, PRC;Georgia Institute of Technology Atlanta, USA;Kyoto University Kyoto, Japan;Purdue University West Lafayette, USA;National Taiwan Normal University Taipei, ROC;University of Liverpool Suzhou, PRC;Hong Kong University of Science;University of Hawaii Honolulu, USA;The University of Texas at Dallas Dallas, USA;University of Wisconsin-Madison Madison, USA;Emory University Atlanta, USA;College of William \& Mary Williamsburg, USA
SelfDefend: LLMs Can Defend Themselves against Jailbreaking in a Practical Manner
The Hong Kong University of Science and Technology(香港科技大学);University of Oregon(俄勒冈大学);Nanyang Technological University(南洋理工大学);City University of Hong Kong(香港城市大学);HSBC(汇丰银行)
Jailbreaking Large Language Models through Iterative Tool-Disguised Attacks via Reinforcement Learning
School of Cyberspace Science and Technology, Beijing Institute of Technology(信息科学与技术学院,北京理工大学);School of Computer Science and Technology, Beijing Institute of Technology(计算机科学与技术学院,北京理工大学);School of Computer Science, University of Auckland(计算机科学学院,奥克兰大学);QAX Security Center, Qi-AnXin Technology Group Inc.(QAX安全中心,启安新科技集团);Qi-AnXin Technology Group Inc.(启安新科技集团);Zhongguancun Laboratory(中关村实验室)
Fuzz-Testing Meets LLM-Based Agents: An Automated and Efficient Framework for Jailbreaking Text-To-Image Generation Models
School of Cyber Science and Technology, Shandong University(山东大学网络科学与技术学院);Netflix Eyeline Studios;State Key Laboratory of Cryptography and Digital Economy Security, Shandong University(山东大学密码与数字经济安全国家重点实验室);Shandong Key Laboratory of Artificial Intelligence Security, Shandong University(山东省人工智能安全重点实验室)