迭代纳什策略优化：通过无悔学习使大语言模型与一般偏好对齐
Iterative Nash Policy Optimization: Aligning LLMs with General Preferences via No-Regret Learning
AI总结：
本文提出了一种基于双人博弈和无悔学习的迭代纳什策略优化（INPO）算法，通过直接在偏好数据集上最小化新损失目标，避免了高昂的胜率估计成本，有效实现了LLMs与一般偏好的对齐。
摘要：
基于人类反馈的强化学习（RLHF）在使大语言模型（LLMs）与人类偏好对齐方面取得了巨大成功。主流的RLHF方法是基于奖励的，遵循Bradley-Terry（BT）模型假设，这可能无法完全捕捉人类偏好的复杂性。在本文中，我们在一般偏好框架下探索RLHF，并从博弈论的角度来处理它。具体来说，我们将问题表述为双人博弈，并提出了一种新颖的在线算法，即迭代纳什策略优化（INPO）。其关键思想是让策略通过无悔学习与自身博弈，从而逼近纳什策略。与以前的方法不同，INPO避免了估计单个响应的预期胜率，这通常会产生高昂的计算或标注成本。相反，我们引入了一个新的损失目标，直接在偏好数据集上进行最小化。我们为该方法提供了理论分析，并通过在各种代表性基准上的实验证明了其有效性。基于LLaMA-3-8B的SFT模型，INPO在AlpacaEval 2.0上实现了42.6%的长度控制胜率，在Arena-Hard上实现了37.8%的胜率，显示出相对于最先进的在线RLHF算法的显著改进。
英文摘要：
Reinforcement Learning with Human Feedback (RLHF) has achieved great success in aligning large language models (LLMs) with human preferences. Prevalent RLHF approaches are reward-based, following the Bradley-Terry (BT) model assumption, which may not fully capture the complexity of human preferences. In this paper, we explore RLHF under a general preference framework and approach it from a game-theoretic perspective. Specifically, we formulate the problem as a two-player game and propose a novel online algorithm, iterative Nash policy optimization (INPO). The key idea is to let the policy play against itself via no-regret learning, thereby approximating the Nash policy. Unlike previous methods, INPO bypasses the need for estimating the expected win rate for individual responses, which typically incurs high computational or annotation costs. Instead, we introduce a new loss objective that is directly minimized over a preference dataset. We provide theoretical analysis for our approach and demonstrate its effectiveness through experiments on various representative benchmarks. With an LLaMA-3-8B-based SFT model, INPO achieves a 42.6% length-controlled win rate on AlpacaEval 2.0 and a 37.8% win rate on Arena-Hard, showing substantial improvement over the state-of-the-art online RLHF algorithms.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
