arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2401.16656cs.CL
收藏

基于梯度的语言模型红队测试

Gradient-Based Language Model Red Teaming

Google Research(谷歌研究院);Anthropic

Nevan Wichers, Carson Denison, Ahmad Beirami

AI总结:

针对人工红队测试语言模型安全弱点难规模化的问题,提出基于梯度的红队测试方法GBRT,通过提示学习及两种变体优化,实验证明其效果优于强化学习基线,且对已微调的安全模型依然有效。

摘要:

红队测试是识别生成式语言模型(LM)弱点的常用策略,即生成能触发LM生成不安全响应的对抗性提示。红队测试对模型对齐和评估都至关重要,但人工开展时耗时费力且难以规模化。本文提出基于梯度的红队测试(GBRT),这是一种自动生成多样化提示、大概率引发LM输出不安全响应的红队方法。GBRT属于提示学习的一种,训练时先用安全分类器对LM响应打分,再通过冻结的安全分类器和LM反向传播以更新提示。为提升输入提示的连贯性,我们提出两种变体:一种添加真实感损失,另一种微调预训练模型来生成提示而非直接学习提示。实验表明,GBRT在寻找能触发LM生成不安全响应的提示方面,比一种强大的基于强化学习的红队方法更有效,且即便LM已被微调以生成更安全的输出,GBRT依然有效。

英文摘要:

Red teaming is a common strategy for identifying weaknesses in generative language models (LMs), where adversarial prompts are produced that trigger an LM to generate unsafe responses. Red teaming is instrumental for both model alignment and evaluation, but is labor-intensive and difficult to scale when done by humans. In this paper, we present Gradient-Based Red Teaming (GBRT), a red teaming method for automatically generating diverse prompts that are likely to cause an LM to output unsafe responses. GBRT is a form of prompt learning, trained by scoring an LM response with a safety classifier and then backpropagating through the frozen safety classifier and LM to update the prompt. To improve the coherence of input prompts, we introduce two variants that add a realism loss and fine-tune a pretrained model to generate the prompts instead of learning the prompts directly. Our experiments show that GBRT is more effective at finding prompts that trigger an LM to generate unsafe responses than a strong reinforcement learning-based red teaming approach, and succeeds even when the LM has been fine-tuned to produce safer outputs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑