对抗偏好优化：通过RM-LLM博弈增强你的对齐
Adversarial Preference Optimization: Enhancing Your Alignment via RM-LLM Game
AI总结：
提出对抗偏好优化（APO）框架，通过LLM与奖励模型的极小极大博弈交替更新，无需额外标注即可适应分布偏移，显著提升对齐基线的有用性与无害性。
摘要：
人类偏好对齐对于提高大型语言模型（LLMs）的交互质量至关重要。现有的对齐方法依赖于人工标注的偏好数据来指导LLM的优化方向。然而，持续更新LLM进行对齐会在模型生成的样本和人工标注的响应之间产生分布差距，阻碍训练效果。为了缓解这一问题，先前的方法需要对新生成的样本进行额外的偏好标注以适应分布偏移，这消耗了大量的标注资源。为了更高效地优化人类偏好，我们提出了一种对抗偏好优化（APO）框架，其中LLM和奖励模型通过极小极大博弈交替更新。通过对抗训练，奖励模型能够无需任何额外标注就适应LLM偏移的生成分布。通过全面的实验，我们发现所提出的对抗训练框架在LLM的有用性和无害性方面进一步增强了现有的对齐基线。代码位于https://github.com/Linear95/APO。
英文摘要：
Human preference alignment is essential to improve the interaction quality of large language models (LLMs). Existing alignment methods depend on manually annotated preference data to guide the LLM optimization directions. However, continuously updating LLMs for alignment raises a distribution gap between model-generated samples and human-annotated responses, hindering training effectiveness. To mitigate this issue, previous methods require additional preference annotation on newly generated samples to adapt to the shifted distribution, which consumes a large amount of annotation resources. Targeting more efficient human preference optimization, we propose an Adversarial Preference Optimization (APO) framework, in which the LLM and the reward model update alternatively via a min-max game. Through adversarial training, the reward model can adapt to the shifted generation distribution of the LLM without any additional annotation. With comprehensive experiments, we find the proposed adversarial training framework further enhances existing alignment baselines in terms of LLM helpfulness and harmlessness. The code is at https://github.com/Linear95/APO.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
