人类反馈可以以不可预测且不希望的方式改变语言模型,因为实践者缺乏对反馈数据编码内容的清晰理解。尽管先前工作研究了某些属性(例如长度或奉承性)的偏好,但自动提取相关特征而无需预先指定假设仍具挑战性。我们介绍了What's In My Human Feedback?(WIMHF),一种使用稀疏自编码器解释反馈数据的方法。WIMHF刻画了(1)数据集能够测量的偏好以及(2)标注者实际表达的偏好。在7个数据集中,WIMHF识别出少量的人类可解释特征,这些特征能够解释黑盒模型获得的大部分偏好预测信号。这些特征揭示了人类偏好的广泛多样性以及数据集层面的上下文作用:例如,Reddit用户偏好非正式性和笑话,而HH-RLHF和PRISM的标注者则不偏好这些。WIMHF还揭示了潜在不安全的偏好,如LMArena用户倾向于投票反对拒绝,通常支持有毒内容。所学特征使数据筛选变得有效:重标记Arena中的有害示例可带来显著的安全收益(+37%),而不会影响通用性能。它们还允许细粒度个性化:在Community Alignment数据集中,我们学习了标注者特定的主观特征权重,以提高偏好预测。WIMHF为从业者提供了以人类为中心的分析方法,以更好地理解和使用偏好数据。
英文摘要:
Human feedback can alter language models in unpredictable and undesirable ways, as practitioners lack a clear understanding of what feedback data encodes. While prior work studies preferences over certain attributes (e.g., length or sycophancy), automatically extracting relevant features without pre-specifying hypotheses remains challenging. We introduce What's In My Human Feedback? (WIMHF), a method to explain feedback data using sparse autoencoders. WIMHF characterizes both (1) the preferences a dataset is capable of measuring and (2) the preferences that the annotators actually express. Across 7 datasets, WIMHF identifies a small number of human-interpretable features that account for the majority of the preference prediction signal achieved by black-box models. These features reveal a wide diversity in what humans prefer, and the role of dataset-level context: for example, users on Reddit prefer informality and jokes, while annotators in HH-RLHF and PRISM disprefer them. WIMHF also surfaces potentially unsafe preferences, such as that LMArena users tend to vote against refusals, often in favor of toxic content. The learned features enable effective data curation: re-labeling the harmful examples in Arena yields large safety gains (+37%) with no cost to general performance. They also allow fine-grained personalization: on the Community Alignment dataset, we learn annotator-specific weights over subjective features that improve preference prediction. WIMHF provides a human-centered analysis method for practitioners to better understand and use preference data.
Interpretability Is Not Just a Tax: Sparsity and Diversity Yield Legible and Composable Reward Experts for Personalization
Saarland University(萨尔兰大学);Independent Researcher(独立研究者);Bielefeld University(比勒菲尔德大学);Max Planck Institute for Software Systems(马克斯·普朗克软件系统研究所);Max Planck Institute for Informatics(马克斯·普朗克信息研究所)
Aligning LLMs through Multi-perspective User Preference Ranking-based Feedback for Programming Question Answering
School of Computer Science and Technology, University of Science and Technology of China(中国科学技术大学计算机科学与技术学院);Hefei University of Technology(合肥工业大学);School of Artificial Intelligence and Data Science, University of Science and Technology of China(中国科学技术大学人工智能与数据科学学院);University of Toronto(多伦多大学)
Aligning Large Language Models from Self-Reference AI Feedback with one General Principle
School of Computer Science, Fudan University(复旦大学计算机科学学院);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);The University of Sydney(悉尼大学);Nanyang Technological University(南洋理工大学)
提出自参考 AI 反馈框架,让 Llama2-Chat 依据通用原则、自我批评与自一致性生成高质量偏好数据,显著提升强化学习对齐效果。
PRISMA: Preference-Reinforced Self-Training Approach for Interpretable Emotionally Intelligent Negotiation Dialogues
Department of Computer Science and Engineering, Indian Institute of Technology Patna, India(印度理工学院帕纳瓦分校计算机科学与工程系);School of Artificial Intelligence and Data Science, Indian Institute of Technology Jodhpur, India(印度理工学院朱道尔分校人工智能与数据科学学院)
Integrating Physician Diagnostic Logic into Large Language Models: Preference Learning from Process Feedback
School of Computer Science, Peking University(北京大学计算机学院);Key Laboratory of High Confidence Software Technologies(PKU), MOE, China(教育部高可信软件技术重点实验室);Beijing Jiaotong University(北京交通大学)
Preference Learning from Physics-Based Feedback: Tuning Language Models to Design BCC/B2 Superalloys
Department of Computer Science, University of New Hampshire(新罕布什尔大学计算机科学系);Materials Department, University of California, Santa Barbara(加州大学圣芭芭拉分校材料系);Department of Mechanical Engineering, University of California, Santa Barbara(加州大学圣芭芭拉分校机械工程系);Allen Institute for Artificial Intelligence(人工智能研究院)
WildFeedback: Aligning LLMs With In-situ User Interactions And Feedback
Microsoft Corporation(微软公司);Purdue University(普渡大学);Texas A&M University(德克萨斯农工大学);University of California San Diego(加州大学圣地亚哥分校);University of Southern California(南加州大学)
RewardExplainer: Learning Reward Model Explanations from Counterfactual Preference Feedback
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Shanghai Jiao Tong University(上海交通大学);Carnegie Mellon University(卡内基梅隆大学);Jilin University(吉林大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
CheXalign: Preference fine-tuning in chest X-ray interpretation models without human feedback
Stanford University(斯坦福大学);University of Oxford(牛津大学);University Hospital Zurich, University of Zurich(苏黎世大学医院、苏黎世大学);Yongin Severance Hospital, Yonsei University(永临松医院、延世大学)