arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2408.02651cs.CLcs.AIcs.CR
收藏

强化学习能否解锁对齐大语言模型中隐藏的危险?

Can Reinforcement Learning Unlock the Hidden Dangers in Aligned Large Language Models?

University of Texas at San Antonio(德克萨斯大学圣安东尼奥分校);Secure AI and Autonomy Lab(安全与自主智能实验室)

Mohammad Bahrami Karkevandi, Nishant Vishwamitra, Peyman Najafirad

AI总结:

本文提出利用强化学习优化对抗性触发器,仅需推理API和小型代理模型即可越狱对齐大语言模型,并通过BERTScore奖励函数提升触发器在黑盒模型上的可迁移性与有效性。

摘要:

大语言模型(LLMs)在自然语言任务中展现出令人印象深刻的能力,但由于其基于互联网文本语料进行训练,其安全性和道德性仍然存在争议。为应对这些担忧,研究者开发了对齐技术,以提升大语言模型的公共可用性和安全性。然而,通过这些模型生成有害内容的可能性似乎依然存在。本文探讨了越狱大语言模型的概念,即通过对抗性触发器逆转其对齐。以往的方法,如软嵌入提示、手工构造提示和基于梯度的自动提示,在黑盒模型上取得的成功有限,因为它们需要模型访问权限,且手工构造提示的多样性较低,容易被拦截。本文提出一种新方法,利用强化学习优化对抗性触发器,仅需目标模型的推理API访问权限和一个较小的代理模型。我们的方法利用基于BERTScore的奖励函数,增强了对抗性触发器在新黑盒模型上的可迁移性和有效性。我们证明,该方法在一个此前未经测试的语言模型上提升了对抗性触发器的性能。

英文摘要:

Large Language Models (LLMs) have demonstrated impressive capabilities in natural language tasks, but their safety and morality remain contentious due to their training on internet text corpora. To address these concerns, alignment techniques have been developed to improve the public usability and safety of LLMs. Yet, the potential for generating harmful content through these models seems to persist. This paper explores the concept of jailbreaking LLMs-reversing their alignment through adversarial triggers. Previous methods, such as soft embedding prompts, manually crafted prompts, and gradient-based automatic prompts, have had limited success on black-box models due to their requirements for model access and for producing a low variety of manually crafted prompts, making them susceptible to being blocked. This paper introduces a novel approach using reinforcement learning to optimize adversarial triggers, requiring only inference API access to the target model and a small surrogate model. Our method, which leverages a BERTScore-based reward function, enhances the transferability and effectiveness of adversarial triggers on new black-box models. We demonstrate that this approach improves the performance of adversarial triggers on a previously untested language model.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑