排列偏好对齐：基于人类判断的列表排序
Permutative Preference Alignment from Listwise Ranking of Human Judgments
AI总结：
针对现有偏好对齐方法在列表排序上的不足，提出基于NDCG的离线列表级排列偏好对齐方法，通过可微替代损失实现端到端训练，在基准上优于成对和列表级方法。
摘要：
将大型语言模型（LLM）与人类偏好对齐对于确保模型行为符合期望且可控至关重要。当前方法，如基于人类反馈的强化学习（RLHF）和直接偏好优化（DPO），依赖Bradley-Terry（B-T）模型来最大化成对选择的似然。然而，当存在多个响应时，B-T模型无法保证响应的准确列表排序。为解决此问题，我们提出排列偏好对齐（PPA），一种新颖的离线列表级方法，将归一化折损累计增益（NDCG）——一种广泛使用的排序指标——作为LLM对齐的替代训练目标。我们通过使用可微的替代损失近似NDCG，开发了一种端到端的对齐算法。实验表明，PPA在评估集和通用基准（如AlpacaEval）上优于现有的成对和列表级方法。此外，我们证明基于NDCG的方法比基于B-T的方法更有效地提高排序准确性，并为这一改进提供了理论解释。
英文摘要：
Aligning Large Language Models (LLMs) with human preferences is crucial in ensuring desirable and controllable model behaviors. Current methods, such as Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO), rely on the Bradley-Terry (B-T) model to maximize the likelihood of pairwise choices. However, when multiple responses are available, the B-T model fails to guarantee an accurate list ranking of the responses. To address this issue, we propose Permutative Preference Alignment (PPA), a novel offline listwise approach that incorporates the Normalized Discounted Cumulative Gain (NDCG), a widely-used ranking metric, as an alternative training objective for LLM alignment. We develop an end-to-end alignment algorithm by approximating NDCG with a differentiable surrogate loss. Experiments demonstrate that PPA outperforms existing pairwise and listwise methods on evaluation sets and general benchmarks such as AlpacaEval. Furthermore, we show that NDCG-based approaches improve ranking accuracy more effectively than B-T-based methods and provide a theoretical explanation for this improvement.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
