Preference-based alignment methods, most prominently Reinforcement Learning with Human Feedback (RLHF), use the judgments of human annotators to shape large language model behaviour. However, the normative role of these judgments is rarely made explicit. I distinguish three conceptual models of that role. The first is extension: annotators extend the system designers' own judgments about what outputs should be. The second is evidence: annotators provide independent evidence about some facts, whether moral, social or otherwise. The third is authority: annotators have some independent authority (as representatives of the broader population) to determine system outputs. I argue that these models have implications for how RLHF pipelines should solicit, validate and aggregate annotations. I survey landmark papers in the literature on RLHF and related methods to illustrate how they implicitly draw on these models, describe failure modes that come from unintentionally or intentionally conflating them, and offer normative criteria for choosing among them. My central recommendation is that RLHF pipeline designers should decompose annotation into separable dimensions and tailor each pipeline to the model most appropriate for that dimension, rather than seeking a single unified pipeline.
CleanRM: Reward Modeling from Noisy Feedback with a Few Clean Feedback for RLHF
Zhejiang University(浙江大学);Xiaohongshu Inc(小红书公司);National University of Singapore(新加坡国立大学);New York University(纽约大学);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学)
The Flip Side of RLHF: On-Policy Feedback for Reward Model Self-Supervised Improvement
State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China(认知智能国家重点实验室,中国科学技术大学);University of Science and Technology of China(中国科学技术大学);Institute of Artificial Intelligence, Hefei Comprehensive National Science Center(合肥综合性国家科学中心人工智能研究院);State Key Laboratory of General Artificial Intelligence, BIGAI(通用人工智能国家重点实验室,BIGAI)
AlignDistil: Token-Level Language Model Alignment as Adaptive Policy Distillation
Key Laboratory of Big Data & Artificial Intelligence in Transportation, (Beijing Jiaotong University), Ministry of Education(大数据与人工智能交通运输联合实验室,(北京交通大学)教育部);School of Computer Science and Technology, Beijing Jiaotong University, Beijing, China(计算机科学与技术学院,北京交通大学,北京,中国);Tencent Inc, China(腾讯公司,中国)
Interpretability Is Not Just a Tax: Sparsity and Diversity Yield Legible and Composable Reward Experts for Personalization
Saarland University(萨尔兰大学);Independent Researcher(独立研究者);Bielefeld University(比勒菲尔德大学);Max Planck Institute for Software Systems(马克斯·普朗克软件系统研究所);Max Planck Institute for Informatics(马克斯·普朗克信息研究所)
Free Lunch Alignment of Text-to-Image Diffusion Models without Preference Image Pairs
University of British Columbia(不列颠哥伦比亚大学);Vector Institute for AI(人工智能向量研究所);Canada CIFAR AI Chair(加拿大CIFAR人工智能主席);Kling Team, Kuaishou Technology(快手科技 Kling 团队);NSERC CRC Chair(加拿大NSERC CRC主席)
Learning From Crowdsourced Noisy Labels: A Signal Processing Perspective
Department of Electrical and Computer Engineering, University of Central Florida(中央佛罗里达大学电气与计算机工程系);School of Electrical Engineering and Computer Science, Oregon State University(俄勒冈州立大学电气工程与计算机科学学院);Department of Electrical and Computer Engineering, University of Minnesota(明尼苏达大学电气与计算机工程系)
Adversarial Preference Learning for Robust LLM Alignment
Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Northeastern University(东北大学);University of Science and Technology of China(中国科学技术大学);MemTensor (Shanghai) Technology Co., Ltd(MemTensor(上海)技术有限公司);University of Auckland(奥克兰大学);China Telecom Corporation Limited Beijing Research Institute(中国电信股份有限公司北京研究院)