arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.10977cs.CLcs.AI
收藏

面向最优大语言模型对齐的双人博弈方法

Toward Optimal LLM Alignments Using Two-Player Games

Fudan University(复旦大学);Northwestern University(西北大学);ByteDance Research(字节跳动研究院)

Rui Zheng, Hongyi Guo, Zhihan Liu, Xiaoying Zhang, Yuanshun Yao, Xiaojun Xu, Zhaoran Wang, Zhiheng Xi, Tao Gui, Qi Zhang, Xuanjing Huang, Hang Li, Yang Liu

AI总结:

本文提出通过对抗性智能体与防御性智能体之间的双人博弈实现大语言模型对齐,理论证明该迭代强化学习优化收敛到纳什均衡,并在安全场景中验证其能提升双方策略的泛化能力。

摘要:

标准的基于人类反馈的强化学习(RLHF)框架主要侧重于利用预先收集的提示来优化大语言模型的性能。然而,收集能够提供全面覆盖的提示既繁琐又具有挑战性,并且往往无法涵盖大语言模型最需要改进的场景。在本文中,我们通过双智能体博弈的视角研究对齐问题,涉及对抗性智能体与防御性智能体之间的迭代交互。对抗性智能体在每一步的任务是生成能够暴露防御性智能体弱点的提示。作为回应,防御性智能体基于奖励模型的反馈,针对这些新识别出的、其难以应对的提示改进自身的回答。我们从理论上证明,这种迭代强化学习优化会收敛到由两个智能体所诱导博弈的纳什均衡。安全场景下的实验结果表明,在这种竞争环境中学习不仅能够充分训练智能体,而且能够使对抗性智能体和防御性智能体都获得具有更强泛化能力的策略。

英文摘要:

The standard Reinforcement Learning from Human Feedback (RLHF) framework primarily focuses on optimizing the performance of large language models using pre-collected prompts. However, collecting prompts that provide comprehensive coverage is both tedious and challenging, and often fails to include scenarios that LLMs need to improve on the most. In this paper, we investigate alignment through the lens of two-agent games, involving iterative interactions between an adversarial and a defensive agent. The adversarial agent's task at each step is to generate prompts that expose the weakness of the defensive agent. In return, the defensive agent seeks to improve its responses to these newly identified prompts it struggled with, based on feedback from the reward model. We theoretically demonstrate that this iterative reinforcement learning optimization converges to a Nash Equilibrium for the game induced by the agents. Experimental results in safety scenarios demonstrate that learning in such a competitive environment not only fully trains agents but also leads to policies with enhanced generalization capabilities for both adversarial and defensive agents.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑