揭示大语言模型中的隐式毒性
Unveiling the Implicit Toxicity in Large Language Models
AI总结：
本文揭示大语言模型能生成难以检测的隐式毒性输出，并提出基于强化学习的攻击方法诱导此类输出，实验证明该方法显著提升攻击成功率，同时所标注样本可增强毒性分类器的检测能力。
摘要：
大语言模型（LLMs）的开放性与其强大的能力相结合，在被用于恶意目的时可能引发新的安全问题。尽管近期研究主要关注探测那些可被现有毒性分类器轻易检测到的毒性输出，我们却发现LLMs能够生成多样化的隐式毒性输出，这些输出仅通过简单的零样本提示极难被检测到。此外，我们提出了一种基于强化学习（RL）的攻击方法，以进一步诱导LLMs中的隐式毒性。具体而言，我们通过一个奖励函数来优化语言模型，该奖励更偏好隐式毒性输出而非显式毒性输出和无毒输出。在五个广泛使用的毒性分类器上的实验表明，通过RL微调可以显著提高攻击成功率。例如，经过RL微调的LLaMA-13B模型在BAD上达到90.04%的攻击成功率，在Davinci003上达到62.85%。我们的发现表明，LLMs在生成不可检测的隐式毒性输出方面构成了重大威胁。我们进一步表明，利用我们攻击方法所标注的样本对毒性分类器进行微调，可以有效增强其检测LLM生成的隐式毒性语言的能力。代码已公开于 https://github.com/thu-coai/Implicit-Toxicity。
英文摘要：
The open-endedness of large language models (LLMs) combined with their impressive capabilities may lead to new safety issues when being exploited for malicious use. While recent studies primarily focus on probing toxic outputs that can be easily detected with existing toxicity classifiers, we show that LLMs can generate diverse implicit toxic outputs that are exceptionally difficult to detect via simply zero-shot prompting. Moreover, we propose a reinforcement learning (RL) based attacking method to further induce the implicit toxicity in LLMs. Specifically, we optimize the language model with a reward that prefers implicit toxic outputs to explicit toxic and non-toxic ones. Experiments on five widely-adopted toxicity classifiers demonstrate that the attack success rate can be significantly improved through RL fine-tuning. For instance, the RL-finetuned LLaMA-13B model achieves an attack success rate of 90.04% on BAD and 62.85% on Davinci003. Our findings suggest that LLMs pose a significant threat in generating undetectable implicit toxic outputs. We further show that fine-tuning toxicity classifiers on the annotated examples from our attacking method can effectively enhance their ability to detect LLM-generated implicit toxic language. The code is publicly available at https://github.com/thu-coai/Implicit-Toxicity.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
