我们应该更听谁的？面向多元对齐的福利感知偏好获取
Who Should We Listen to More? Welfare-Aware Preference Acquisition for Pluralistic Alignment
AI总结：
本研究针对多元对齐中人类反馈分配问题，提出WAPA方法，通过福利目标分配反馈，实验显示其可提升监督效果，20%人类标注可恢复75%-86%的效用差距。
摘要：
多元对齐需要在有限的人类反馈下容纳多样化偏好。使用AI模拟器标注偏好比较可降低标注成本，但不同个体的模拟精度差异会导致部分人的偏好在训练中未被充分代表。本研究探讨如何分配人类反馈以构建多元对齐的训练数据，以及该分配如何影响训练出的策略。为使分配明确且可控，我们定义了基于个体监督效用的社会福利目标，该效用用于衡量标注质量。每个查询的边际福利增益取决于其预期监督改进、接收者当前效用以及应用指定的优先级。福利感知偏好获取（WAPA）使用辅助参考模型估计查询增益和个体效用，随后根据估计的边际福利增益选择查询并更新接收者效用。我们证明了在固定查询预算和模型预测下，该获取算法对估计的福利目标具有精确最优性。在6个角色扮演（Roleplay）集合和4个观点问答（OpinionQA）区域的实验表明，监督效果和可控分配权衡得到提升。对4个角色扮演集合的策略训练提高了对可见和不可见个体的偏好预测。当优先考虑指定群体和代表性不足的个体时，获取20%的比较的人类标注可恢复完全基于AI生成标签训练的策略与完全基于真实标签训练的策略之间平均预测效用差距的75%-86%。
英文摘要：
Pluralistic alignment needs to accommodate diverse preferences under limited human feedback. Using AI simulators to label preference comparisons reduces annotation costs, but variation in simulation accuracy across individuals can leave some people's preferences poorly represented in training. We study how to allocate human feedback across people to construct training data for pluralistic alignment, and how this allocation affects the trained policy. To make this allocation explicit and controllable, we define a social welfare objective over individual supervision utilities that measure label quality. Each query's marginal welfare gain depends on its expected supervision improvement, the recipient's current utility, and application-designated priority. Welfare-Aware Preference Acquisition (WAPA) uses auxiliary reference models to estimate query gains and individual utilities, then selects queries by estimated marginal welfare gain while updating recipient utilities. We prove exact optimality of the acquisition algorithm for the estimated welfare objective under a fixed query budget and model predictions. Experiments on six Roleplay collections and four OpinionQA regions demonstrate improved supervision and controllable allocation tradeoffs. Policy training on four Roleplay collections improves preference prediction for both seen and unseen individuals. When prioritizing a designated group and poorly represented individuals, acquiring human labels for 20% of comparisons recovers 75-86% of the average predictive utility gap between policies trained entirely on AI-generated labels and entirely on ground-truth labels.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
