纠正KL正则化的迷思：通过卡方偏好优化实现无过度优化的直接对齐
Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization
AI总结：
针对RLHF等方法的过度优化问题，提出仅修改DPO对数链接函数的χPO算法，通过卡方散度正则化缓解过度优化，获单策略集中度样本复杂度保证，成首个可证明鲁棒的实用通用离线对齐算法。
摘要：
人类反馈强化学习（RLHF）等语言模型对齐方法显著提升了语言模型能力，但受限于广泛观察到的过度优化现象——对齐过程中语言模型质量下降。模型针对离线奖励模型优化性能时，会过度拟合误差并偏离数据涵盖的偏好响应。现有离线对齐方法广泛采用KL正则化抑制分布偏移，但过度优化仍损害性能。为从理论上洞察这些实证观察的根源，我们首先证明KL正则化不足以防止过度拟合，进而提出问题：能否设计一种可证明对过度优化鲁棒的高效算法？我们提出离线对齐新算法卡方偏好优化（χPO）以解决该问题。χPO是对直接偏好优化（DPO；Rafailov等，2023）的一行修改，仅调整DPO目标中的对数链接函数。尽管修改极小，χPO通过卡方散度正则化隐式实现不确定性下的悲观原则——卡方散度比KL正则化更有效量化不确定性——并可证明缓解过度优化，基于单策略集中度（离线强化学习的黄金标准）实现样本复杂度保证。χPO的简洁性和强保证使其成为首个可证明对过度优化鲁棒的实用通用离线对齐算法。
英文摘要：
Language model alignment methods such as reinforcement learning from human feedback (RLHF) have led to impressive advances in language model capabilities, but are limited by a widely observed phenomenon known as overoptimization, where the quality of the language model degrades over the course of the alignment process. As the model optimizes performance with respect to an offline reward model, it overfits to inaccuracies and drifts away from preferred responses covered by the data. To discourage such distribution shift, KL-regularization is widely employed in existing offline alignment methods, but overoptimization continues to harm performance. Lending theoretical insight into the source of these empirical observations, we first show that the KL-regularization is too weak to prevent overfitting, then raise the following question: is it possible to design an efficient algorithm that is provably robust to overoptimization? We address this question with a new algorithm for offline alignment, $χ^2$-Preference Optimization ($χ$PO). $χ$PO is a one-line change to Direct Preference Optimization (DPO; Rafailov et al., 2023), which only involves modifying the logarithmic link function in the DPO objective. Despite this minimal change, $χ$PO implicitly implements the principle of pessimism in the face of uncertainty via regularization with the $χ^2$-divergence -- which quantifies uncertainty more effectively than KL-regularization -- and provably alleviates overoptimization, achieving sample-complexity guarantees based on single-policy concentrability -- the gold standard in offline reinforcement learning. $χ$PO's simplicity and strong guarantees make it the first practical and general-purpose offline alignment algorithm that is provably robust to overoptimization.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
