Large Language Models (LLMs) face threats from jailbreak prompts. Existing methods for detecting jailbreak prompts are primarily online moderation APIs or finetuned LLMs. These strategies, however, often require extensive and resource-intensive data collection and training processes. In this study, we propose GradSafe, which effectively detects jailbreak prompts by scrutinizing the gradients of safety-critical parameters in LLMs. Our method is grounded in a pivotal observation: the gradients of an LLM's loss for jailbreak prompts paired with compliance response exhibit similar patterns on certain safety-critical parameters. In contrast, safe prompts lead to different gradient patterns. Building on this observation, GradSafe analyzes the gradients from prompts (paired with compliance responses) to accurately detect jailbreak prompts. We show that GradSafe, applied to Llama-2 without further training, outperforms Llama Guard, despite its extensive finetuning with a large dataset, in detecting jailbreak prompts. This superior performance is consistent across both zero-shot and adaptation scenarios, as evidenced by our evaluations on ToxicChat and XSTest. The source code is available at https://github.com/xyq7/GradSafe.
Cross-modality Information Check for Detecting Jailbreaking in Multimodal Large Language Models
School of Information Science and Technology, ShanghaiTech University(上海科技大学信息科学与技术学院);The State Key Laboratory of Blockchain and Data Security, Zhejiang University(浙江大学区块链与数据安全国家重点实验室)
School of Information Management, Wuhan University(武汉大学信息管理学院);Nanyang Technological University(南洋理工大学);Torr Vision Group, University of Oxford(牛津大学Torr视觉组)
What Intermediate Layers Know: Detecting Jailbreaks from Entropy Dynamics
LMU Munich(慕尼黑大学);relAI – Konrad Zuse School of Excellence in Reliable AI(relAI – 康拉德·楚泽可靠人工智能卓越学校);University of Pisa(比萨大学);Munich Center for Machine Learning(慕尼黑机器学习中心);School of Engineering, Computing and Mathematics, Oxford Brookes University(牛津布鲁克斯大学工程、计算与数学学院)
Divide and Conquer: A Hybrid Strategy Defeats Multimodal Large Language Models
School of Software, Henan University(河南大学软件学院);Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);University of Chinese Academy of Sciences(中国科学院大学)
SafeDialBench: A Fine-Grained Safety Evaluation Benchmark for Large Language Models in Multi-Turn Dialogues with Diverse Jailbreak Attacks
National Key Laboratory for Novel Software Technology, Nanjing University(南京大学新型软件技术国家重点实验室);University of Liverpool(利物浦大学);City University of Hong Kong(香港城市大学);School of Intelligence Science and Technology, Nanjing University(南京大学智能科学与技术学院);China Mobile Research Institute(中国移动研究院);China Mobile (Suzhou) Software Technology Co., Ltd(中国移动苏州软件技术有限公司)
Feature-Aware Malicious Output Detection and Mitigation
Tianjin University(天津大学);Meituan Group(美团集团);Tsinghua University(清华大学);Aerospace Information Research Institute, Chinese Academy of Sciences(中国科学院空天信息创新研究院);Monash University(莫纳什大学)
Copyright Detective: A Forensic System to Evidence LLMs Flickering Copyright Leakage Risks
Pine AI;The University of Texas at Austin(德克萨斯大学奥斯汀分校);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Duke University(杜克大学);University of Michigan(密歇根大学);Stevens Institute of Technology(史蒂文斯理工学院);Johns Hopkins University(约翰霍普金斯大学);University of Notre Dame(圣母大学);The Pennsylvania State University(宾夕法尼亚州立大学)
RigorLLM: Resilient Guardrails for Large Language Models against Undesired Content
University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Virginia Tech(弗吉尼亚理工大学);Salesforce Research(Salesforce研究院);University of California Berkeley(加州大学伯克利分校);University of Chicago(芝加哥大学)
Leveraging the Potential of Prompt Engineering for Hate Speech Detection in Low-Resource Languages
Bangladesh Institute of Governance and Management (BIGM)(孟加拉国治理与管理研究所);Department of Cyber Security Engineering, George Mason University(网络安全工程系,乔治·梅森大学)