COMAL：一种用于将大语言模型与一般偏好对齐的收敛元算法
COMAL: A Convergent Meta-Algorithm for Aligning LLMs with General Preferences
AI总结：
COMAL是一种受博弈论启发的元算法，通过将偏好对齐建模为零和博弈并收敛至纳什均衡，确保50%胜率，在多种数据集上优于现有方法，并保持高胜率。
摘要：
许多对齐方法，包括基于人类反馈的强化学习（RLHF），依赖于Bradley-Terry奖励假设，该假设并不总是足以捕捉一般人类偏好的全部范围和复杂性。我们在一般偏好框架下探索RLHF，通过将对齐问题建模为博弈论框架中的两人零和博弈，其中纳什均衡策略保证对任何竞争策略具有50%的胜率。然而，先前用于寻找纳什策略的自对弈算法即使在简单的合成设置中也会发散或仅收敛到修改后博弈中的纳什策略，从而无法保持对所有其他策略的50%胜率保证。我们提出了一种元算法，即收敛元对齐算法（COMAL），用于具有一般偏好的语言模型对齐，其灵感来自博弈论中的收敛算法。我们提供了理论分析，表明我们的元算法在最后一次迭代中收敛到精确的纳什策略，并在一系列合成和偏好优化数据集上证明了其有效性。COMAL简单易用，可以以最小的更改与许多现有的偏好优化方法集成，并且在受控评估中，当应用于Llama-3-8B-Instruct和Qwen2.5-7B时，它相对于所有比较算法始终保持着高于60.2%和56.8%的胜率。
英文摘要：
Many alignment methods, including reinforcement learning from human feedback (RLHF), rely on the Bradley-Terry reward assumption, which is not always sufficient to capture the full range and complexity of general human preferences. We explore RLHF under a general preference framework by modeling the alignment problem as a two-player zero-sum game in a game-theoretic framework, where the Nash equilibrium policy guarantees a 50% win rate against any competing policy. However, previous self-play algorithms for finding the Nash policy either diverge or only converge to a Nash policy in a modified game, even in a simple synthetic setting, thereby failing to maintain the 50% win rate guarantee against all other policies. We propose a meta-algorithm, Convergent Meta Alignment Algorithm (COMAL), for language model alignment with general preferences, inspired by convergent algorithms in game theory. We provide theoretical analysis that our meta-algorithm converges to an exact Nash policy in the last iterate and demonstrate its effectiveness on a range of synthetic and preference optimization datasets. COMAL is simple and can be integrated with many existing methods designed for preference optimization with minimal changes, and empirically it consistently maintains above 60.2% and 56.8% win rates, when applied to Llama-3-8B-Instruct and Qwen2.5-7B, against all compared algorithms under controlled evaluations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
