技巧包：大型语言模型越狱攻击的基准测试
Bag of Tricks: Benchmarking of Jailbreak Attacks on LLMs
AI总结：
本研究提出JailTrickBench基准，系统评估越狱攻击的八个关键因素，并在防御增强型LLM上进行大规模实验，强调标准化评估的重要性。
摘要：
尽管大型语言模型（LLMs）已展现出在零样本方式下执行复杂任务的显著能力，但它们容易受到越狱攻击，并可能被操纵以产生有害输出。近年来，越来越多的研究将越狱攻击分为令牌级和提示级攻击。然而，先前的工作主要忽视了越狱攻击的多种关键因素，大多数研究集中于LLM的脆弱性，而缺乏对防御增强型LLM的探索。为解决这些问题，我们引入了JailTrickBench，以评估各种攻击设置对LLM性能的影响，并为越狱攻击提供基线，鼓励采用标准化的评估框架。具体而言，我们从目标级和攻击级两个视角评估了在LLM上实施越狱攻击的八个关键因素。我们进一步在六个防御方法上进行了七种代表性越狱攻击，使用了两个广泛使用的数据集，涵盖了约354个实验，在A800-80G上花费了约55,000个GPU小时。我们的实验结果强调了需要标准化基准来评估这些攻击对防御增强型LLM的影响。我们的代码可在https://github.com/usail-hkust/JailTrickBench获取。
英文摘要：
Although Large Language Models (LLMs) have demonstrated significant capabilities in executing complex tasks in a zero-shot manner, they are susceptible to jailbreak attacks and can be manipulated to produce harmful outputs. Recently, a growing body of research has categorized jailbreak attacks into token-level and prompt-level attacks. However, previous work primarily overlooks the diverse key factors of jailbreak attacks, with most studies concentrating on LLM vulnerabilities and lacking exploration of defense-enhanced LLMs. To address these issues, we introduced $\textbf{JailTrickBench}$ to evaluate the impact of various attack settings on LLM performance and provide a baseline for jailbreak attacks, encouraging the adoption of a standardized evaluation framework. Specifically, we evaluate the eight key factors of implementing jailbreak attacks on LLMs from both target-level and attack-level perspectives. We further conduct seven representative jailbreak attacks on six defense methods across two widely used datasets, encompassing approximately 354 experiments with about 55,000 GPU hours on A800-80G. Our experimental results highlight the need for standardized benchmarking to evaluate these attacks on defense-enhanced LLMs. Our code is available at https://github.com/usail-hkust/JailTrickBench.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
