超越RLHF和NLHF：在公理框架下的人口比例对齐
Beyond RLHF and NLHF: Population-Proportional Alignment under an Axiomatic Framework
AI总结：
本文提出了一种基于公理框架的人口比例对齐方法，通过社会选择理论解决传统偏好学习中的偏差和操纵问题，并在推荐任务和语言模型对齐中验证了其有效性。
摘要：
传统偏好学习方法在聚合多个评估者的偏好时往往优先考虑被广泛持有的意见。这可能导致偏袒某些类型意见或群体的政策，并容易受到战略操纵。为了解决这个问题，我们开发了一种新的偏好学习框架，能够将聚合意见和政策与评估者偏好的真实人口分布成比例对齐。基于社会选择理论，我们的方法直接从成对比较数据推断出评估者人口分布的可行集。利用这些估计值，算法构建了一个满足社会选择理论基础公理（即单调性和帕累托效率）以及我们新引入的人口比例对齐和人口有界可操纵性公理的政策。此外，我们提出了一种软max放松方法，可以平滑地在人口比例对齐与选择康多塞胜者（在成对比较中击败所有其他选项）之间进行权衡。最后，我们通过表格推荐任务和大语言模型对齐的实验验证了我们方法的有效性和可扩展性。
英文摘要：
Conventional preference learning methods often prioritize opinions held more widely when aggregating preferences from multiple evaluators. This may result in policies that are biased in favor of some types of opinions or groups and susceptible to strategic manipulation. To address this issue, we develop a novel preference learning framework capable of aligning aggregate opinions and policies proportionally with the true population distribution of evaluator preferences. Grounded in social choice theory, our approach infers the feasible set of evaluator population distributions directly from pairwise comparison data. Using these estimates, the algorithm constructs a policy that satisfies foundational axioms from social choice theory, namely monotonicity and Pareto efficiency, as well as our newly-introduced axioms of population-proportional alignment and population-bounded manipulability. Moreover, we propose a soft-max relaxation method that smoothly trades off population-proportional alignment with the selection of the Condorcet winner (which beats all other options in pairwise comparisons). Finally, we validate the effectiveness and scalability of our approach through experiments on both tabular recommendation tasks and large language model alignment.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
