arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.26202cs.CLcs.AI
收藏

我的人类反馈中有什么?通过稀疏自编码器学习可解释的偏好数据描述

What's In My Human Feedback? Learning Interpretable Descriptions of Preference Data

UC Berkeley(加州大学伯克利分校);FAIR at Meta(Meta AI研究院)

Rajiv Movva, Smitha Milli, Sewon Min, Emma Pierson

AI总结:

本文提出WIMHF方法,通过稀疏自编码器解释人类反馈数据,揭示偏好数据中的人类可解释特征,发现不同数据集中的偏好差异及潜在不安全偏好,为数据筛选和个性化调整提供支持。

摘要:

人类反馈可以以不可预测且不希望的方式改变语言模型,因为实践者缺乏对反馈数据编码内容的清晰理解。尽管先前工作研究了某些属性(例如长度或奉承性)的偏好,但自动提取相关特征而无需预先指定假设仍具挑战性。我们介绍了What's In My Human Feedback?(WIMHF),一种使用稀疏自编码器解释反馈数据的方法。WIMHF刻画了(1)数据集能够测量的偏好以及(2)标注者实际表达的偏好。在7个数据集中,WIMHF识别出少量的人类可解释特征,这些特征能够解释黑盒模型获得的大部分偏好预测信号。这些特征揭示了人类偏好的广泛多样性以及数据集层面的上下文作用:例如,Reddit用户偏好非正式性和笑话,而HH-RLHF和PRISM的标注者则不偏好这些。WIMHF还揭示了潜在不安全的偏好,如LMArena用户倾向于投票反对拒绝,通常支持有毒内容。所学特征使数据筛选变得有效:重标记Arena中的有害示例可带来显著的安全收益(+37%),而不会影响通用性能。它们还允许细粒度个性化:在Community Alignment数据集中,我们学习了标注者特定的主观特征权重,以提高偏好预测。WIMHF为从业者提供了以人类为中心的分析方法,以更好地理解和使用偏好数据。

英文摘要:

Human feedback can alter language models in unpredictable and undesirable ways, as practitioners lack a clear understanding of what feedback data encodes. While prior work studies preferences over certain attributes (e.g., length or sycophancy), automatically extracting relevant features without pre-specifying hypotheses remains challenging. We introduce What's In My Human Feedback? (WIMHF), a method to explain feedback data using sparse autoencoders. WIMHF characterizes both (1) the preferences a dataset is capable of measuring and (2) the preferences that the annotators actually express. Across 7 datasets, WIMHF identifies a small number of human-interpretable features that account for the majority of the preference prediction signal achieved by black-box models. These features reveal a wide diversity in what humans prefer, and the role of dataset-level context: for example, users on Reddit prefer informality and jokes, while annotators in HH-RLHF and PRISM disprefer them. WIMHF also surfaces potentially unsafe preferences, such as that LMArena users tend to vote against refusals, often in favor of toxic content. The learned features enable effective data curation: re-labeling the harmful examples in Arena yields large safety gains (+37%) with no cost to general performance. They also allow fine-grained personalization: on the Community Alignment dataset, we learn annotator-specific weights over subjective features that improve preference prediction. WIMHF provides a human-centered analysis method for practitioners to better understand and use preference data.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑