arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.01461cs.CL
收藏

通过强化学习驱动的查询优化增强大型语言模型的能力与鲁棒性

Enhancing the Capability and Robustness of Large Language Models through Reinforcement Learning-Driven Query Refinement

School of Computer Science, Fudan University, Shanghai, China(复旦大学计算机科学学院)

Xiaohua Wang, Zisu Huang, Feiran Zhang, Zhibo Xu, Cenyuan Zhang, Qi Qian, Xiaoqing Zheng, Xuanjing Huang

AI总结:

本研究提出一种基于强化学习的可迁移且可插拔的查询优化框架,在用户提示输入LLMs前进行优化,既提升了回复质量,又增强了对越狱攻击的鲁棒性。

摘要:

大型语言模型(LLMs)生成诚实、无害且有用回复的能力严重依赖于用户提示的质量。然而,这些提示往往简短且模糊,从而显著限制了LLMs的全部潜力。此外,有害提示可能被对手精心制作和操纵以越狱LLMs,诱导它们产生潜在的有毒内容。为了在增强LLMs能力的同时保持对有害越狱输入的强鲁棒性,本研究提出了一个可迁移且可插拔的框架,在用户提示输入LLMs之前对其进行优化。该策略提高了查询质量,使LLMs能够生成更真实、良性且有用的回复。具体而言,引入了一个轻量级查询优化模型,并使用一种特别设计的强化学习方法进行训练,该方法结合了多个目标以增强LLMs的特定能力。大量实验表明,该优化模型不仅提高了回复质量,还增强了其对越狱攻击的鲁棒性。代码可在以下网址获取:https://github.com/Huangzisu/query-refinement 。

英文摘要:

The capacity of large language models (LLMs) to generate honest, harmless, and helpful responses heavily relies on the quality of user prompts. However, these prompts often tend to be brief and vague, thereby significantly limiting the full potential of LLMs. Moreover, harmful prompts can be meticulously crafted and manipulated by adversaries to jailbreak LLMs, inducing them to produce potentially toxic content. To enhance the capabilities of LLMs while maintaining strong robustness against harmful jailbreak inputs, this study proposes a transferable and pluggable framework that refines user prompts before they are input into LLMs. This strategy improves the quality of the queries, empowering LLMs to generate more truthful, benign and useful responses. Specifically, a lightweight query refinement model is introduced and trained using a specially designed reinforcement learning approach that incorporates multiple objectives to enhance particular capabilities of LLMs. Extensive experiments demonstrate that the refinement model not only improves the quality of responses but also strengthens their robustness against jailbreak attacks. Code is available at: https://github.com/Huangzisu/query-refinement .

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑