JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models
University of Pennsylvania(宾夕法尼亚大学);ETH Zurich(苏黎世联邦理工学院);EPFL(洛桑联邦理工学院);Sony AI(索尼人工智能)
Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J. Pappas,… 展开作者
Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J. Pappas, Florian Tramer, Hamed Hassani, Eric Wong
备注:
The camera-ready version of JailbreakBench v1.0 (accepted at NeurIPS 2024 Datasets and Benchmarks Track): more attack artifacts, more test-time defenses, a more accurate jailbreak judge (Llama-3-70B with a custom prompt), a larger dataset of human preferences for selecting a jailbreak judge (300 examples), an over-refusal evaluation dataset, a semantic refusal judge based on Llama-3-8B
Jailbreak attacks cause large language models (LLMs) to generate harmful, unethical, or otherwise objectionable content. Evaluating these attacks presents a number of challenges, which the current collection of benchmarks and evaluation techniques do not adequately address. First, there is no clear standard of practice regarding jailbreaking evaluation. Second, existing works compute costs and success rates in incomparable ways. And third, numerous works are not reproducible, as they withhold adversarial prompts, involve closed-source code, or rely on evolving proprietary APIs. To address these challenges, we introduce JailbreakBench, an open-sourced benchmark with the following components: (1) an evolving repository of state-of-the-art adversarial prompts, which we refer to as jailbreak artifacts; (2) a jailbreaking dataset comprising 100 behaviors -- both original and sourced from prior work (Zou et al., 2023; Mazeika et al., 2023, 2024) -- which align with OpenAI's usage policies; (3) a standardized evaluation framework at https://github.com/JailbreakBench/jailbreakbench that includes a clearly defined threat model, system prompts, chat templates, and scoring functions; and (4) a leaderboard at https://jailbreakbench.github.io/ that tracks the performance of attacks and defenses for various LLMs. We have carefully considered the potential ethical implications of releasing this benchmark, and believe that it will be a net positive for the community.
BAIT: Boundary-Guided Disclosure Escalation LLM Jailbreaking via Self-Conditioned Reasoning
The Harbin Institute of Technology(哈尔滨工业大学);The Hong Kong Polytechnic University(香港理工大学);Shenzhen University(深圳大学);Shenzhen Loop Area Institute(深圳河套学院)
TokenProber: Jailbreaking Text-to-image Models via Fine-grained Word Impact Analysis
School of Cyber Science and Engineering, Xi’an Jiaotong University(网络安全科学与工程学院,西安交通大学);Singapore Management University(新加坡管理学院);Nanyang Technological University(南洋理工大学)
Scalable Defense against In-the-wild Jailbreaking Attacks with Safety Context Retrieval
School of EECS, Peking University(电子工程系,北京大学);School of Mathematical Sciences, Peking University(数学系,北京大学);State Key Lab of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University(通用人工智能国家重点实验室,智能科学与技术学校,北京大学);Institute for Artificial Intelligence, Peking University(人工智能研究院,北京大学)
In Vino Veritas and Vulnerabilities: Examining LLM Safety via Drunk Language Inducement
School of Computer Science and Engineering, UNSW Sydney(计算机科学与工程学院,新南威尔士大学悉尼分校);School of Computing and Information System, the University of Melbourne(计算与信息系统学院,墨尔本大学)
Data to Defense: The Role of Curation in Customizing LLMs Against Jailbreaking Attacks
Stony Brook University(石溪大学);Binghamton University(宾汉姆顿大学);University of Iowa(爱荷华大学);Dartmouth College(达特茅斯学院);Georgia Institute of Technology(佐治亚理工学院)
AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents
Gray Swan AI(格雷天鹅人工智能公司);UK AI Security Institute(英国人工智能安全研究所);EPFL(洛桑联邦理工学院);Center for AI Safety(人工智能安全中心);Carnegie Mellon University(卡内基梅隆大学);University of Oxford(牛津大学)
Jailbreaking and Mitigation of Vulnerabilities in Large Language Models
Hunan University Changsha, PRC;Georgia Institute of Technology Atlanta, USA;Kyoto University Kyoto, Japan;Purdue University West Lafayette, USA;National Taiwan Normal University Taipei, ROC;University of Liverpool Suzhou, PRC;Hong Kong University of Science;University of Hawaii Honolulu, USA;The University of Texas at Dallas Dallas, USA;University of Wisconsin-Madison Madison, USA;Emory University Atlanta, USA;College of William \& Mary Williamsburg, USA
PolyJailbreak: Cross-Modal Jailbreaking Attacks on Black-Box Multimodal LLMs
School of Cyber Science and Engineering, Sichuan University(四川大学网络空间安全学院);School of Cyber Security, Tianjin University(天津大学网络安全学院);College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院)
HTS-Attack: Heuristic Token Search for Jailbreaking Text-to-Image Models
Agency for Science, Technology and Research (A*STAR)(新加坡科技研究局);CFAR;IHPC;Mohammed Bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Nanyang Technological University(南洋理工大学);Alibaba Group(阿里巴巴集团);University of Oxford(牛津大学)
Why Does Weak-OOD Help? A Further Step Towards Understanding Jailbreaking VLMs
Tsinghua Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);ByteDance(字节跳动);Peking University(北京大学);Nanyang Technological University(南洋理工大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);College of Computer Science and Software Engineering, Shenzhen University(深圳大学计算机科学与软件工程学院)
SelfDefend: LLMs Can Defend Themselves against Jailbreaking in a Practical Manner
The Hong Kong University of Science and Technology(香港科技大学);University of Oregon(俄勒冈大学);Nanyang Technological University(南洋理工大学);City University of Hong Kong(香港城市大学);HSBC(汇丰银行)
Feint and Attack: Attention-Based Strategies for Jailbreaking and Protecting LLMs
Beijing University of Posts and Telecommunications(北京邮电大学);Hangzhou Dianzi University(杭州电子科技大学);Beijing Academy of Artificial Intelligence(北京人工智能研究院);Fujian Cancer Hospital(福建癌症医院)
Mitsubishi Electric Corporation(三菱电机株式会社);Pennsylvania State University(宾夕法尼亚州立大学);The Ohio State University(俄亥俄州立大学);Mitsubishi Electric Research Laboratories(三菱电机研究所)
Adversarial Attacks of Vision Tasks in the Past 10 Years: A Survey
Nanjing University of Aeronautics and Astronautics(南京航空航天大学);The Chinese University of Hong Kong(香港中文大学);The University of Hong Kong(香港大学);Zhejiang Lab, Nanjing University of Aeronautics and Astronautics(浙江实验室,南京航空航天大学)
Mitigating Many-shot Jailbreak Attacks with One Single Demonstration
Zhejiang University(浙江大学);Nanyang Technological University(南洋理工大学);Sun Yat-sen University(中山大学);University of Science and Technology of China(中国科学技术大学);Virginia Tech(弗吉尼亚理工大学)
LLMs Caught in the Crossfire: Malware Requests and Jailbreak Challenges
Institute of Artificial Intelligence (TeleAI), China Telecom(人工智能研究院(TeleAI),中国电信);Beihang University(北京航空航天大学);Beijing Jiaotong University(北京交通大学);Northwestern Polytechnical University(西北工业大学)
DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models
University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Stanford University(斯坦福大学);University of California, Berkeley(加州大学伯克利分校);Center for AI Safety(AI安全中心);Microsoft Corporation(微软公司);The Chinese University of Hong Kong(香港中文大学)