通过排序加权RLHF实现民主偏好对齐
Democratic Preference Alignment via Sortition-Weighted RLHF
AI总结:
DemPO通过算法排序机制优化偏好对齐,确保民主代表性,提升模型对代表性公众价值观的反映能力。
摘要:
人工智能系统应该学习谁的价值观?基于偏好的对齐方法如RLHF的训练信号来自人类评价者,但这些评价者群体通常是方便样本,系统性地过度代表某些人口统计数据,而低估其他人。我们介绍了民主偏好优化(DemPO),一种应用算法排序机制,即用于构建公民议会的机制,用于基于偏好的微调。DemPO提供两种训练方案。硬面板仅在通过排序选出的满足配额的迷你公众的偏好上进行训练。软面板保留所有数据,但通过排序彩票下的包含概率重新加权每个评价者。我们证明软面板加权可以闭合形式地恢复硬面板目标。使用一个公共偏好数据集,该数据集将人类判断与评价者人口统计数据配对,并且一个独立于美国代表性公民议会的75条款宪法,我们评估了从十亿到八十亿参数的Llama模型在每种方案下的微调结果。在六种聚合方法中,硬面板始终排名第一,而软面板始终优于无加权基线,随着模型容量的增加,效果大小增长。这些结果表明,在偏好收集阶段强制人口统计数据代表性,而不是事后修正,会产生行为更能反映代表性公众所提取的价值观的模型。
英文摘要:
Whose values should AI systems learn? Preference based alignment methods like RLHF derive their training signal from human raters, yet these rater pools are typically convenience samples that systematically over represent some demographics and under represent others. We introduce Democratic Preference Optimization, or DemPO, a framework that applies algorithmic sortition, the same mechanism used to construct citizen assemblies, to preference based fine tuning. DemPO offers two training schemes. Hard Panel trains exclusively on preferences from a quota satisfying mini public sampled via sortition. Soft Panel retains all data but reweights each rater by their inclusion probability under the sortition lottery. We prove that Soft Panel weighting recovers the expected Hard Panel objective in closed form. Using a public preference dataset that pairs human judgments with rater demographics and a seventy five clause constitution independently elicited from a representative United States panel, we evaluate Llama models from one billion to eight billion parameters fine tuned under each scheme. Across six aggregation methods, the Hard Panel consistently ranks first and the Soft Panel consistently outperforms the unweighted baseline, with effect sizes growing as model capacity increases. These results demonstrate that enforcing demographic representativeness at the preference collection stage, rather than post hoc correction, yields models whose behavior better reflects values elicited from representative publics.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。