超越成对：通过排名选择建模增强大语言模型对齐
Beyond Pairwise: Empowering LLM Alignment With Ranked Choice Modeling
AI总结：
本文提出RCPO框架，通过最大似然估计结合排名选择建模，提升大语言模型对齐效果，实验显示其在多种模型和设置中均优于基线方法。
摘要：
大语言模型（LLM）的对齐主要依赖于成对偏好优化，其中标注者在两个响应之间选择更好的一个。尽管简单，这种方法忽略了从更丰富的形式的人类反馈中学习的机会，如多向比较和前k排名。我们引入了排名选择偏好优化（RCPO），这是一个统一的框架，通过最大似然估计将偏好优化与（排名）选择建模联系起来。RCPO支持基于效用和基于排名的模型，涵盖了几种成对方法（如DPO和SimPO）作为特殊情况，并为更丰富的反馈格式提供了原则性的训练目标。我们用两个代表模型（多项式对数和Mallows-RMJ）实例化了这个框架。在Llama-3-8B-Instruct、Gemma-2-9B-it和Mistral-7B-Instruct上进行的实验显示，RCPO在分布内和分布外设置中均优于竞争基线。RCPO表明，直接利用排名偏好数据，结合合适的选型模型，可以产生更有效的对齐。它为将（排名）选择建模纳入LLM训练提供了扩展的基础。
英文摘要：
Alignment of large language models (LLMs) has predominantly relied on pairwise preference optimization, where annotators select the better of two responses to a prompt. While simple, this approach overlooks the opportunity to learn from richer forms of human feedback, such as multiway comparisons and top-$k$ rankings. We introduce Ranked Choice Preference Optimization (RCPO), a unified framework that bridges preference optimization with (ranked) choice modeling via maximum likelihood estimation. RCPO supports both utility-based and rank-based models, subsumes several pairwise methods (such as DPO and SimPO) as special cases, and provides principled training objectives for richer feedback formats. We instantiate this framework with two representative models (Multinomial Logit and Mallows-RMJ). Experiments on Llama-3-8B-Instruct, Gemma-2-9B-it, and Mistral-7B-Instruct across in-distribution and out-of-distribution settings show that RCPO consistently outperforms competitive baselines. RCPO shows that directly leveraging ranked preference data, combined with the right choice models, yields more effective alignment. It offers an extensible foundation for incorporating (ranked) choice modeling into LLM training.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
