arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.25895cs.CYcs.AIcs.CL
收藏

三种RLHF注释模型:扩展、证据与权威

Three Models of RLHF Annotation: Extension, Evidence, and Authority

University of Toronto(多伦多大学)

Steve Coyne

AI总结:

本文探讨RLHF注释的三种模型:扩展、证据与权威,分析其对注释流程设计的影响,并提出应根据不同维度选择适配的模型。

摘要:

基于偏好对齐的方法,尤其是人类反馈强化学习(RLHF),利用人类注释者的判断来塑造大语言模型的行为。然而,这些判断的规范作用 rarely 明确。本文区分了三种概念模型:扩展(注释者扩展系统设计者对输出的判断)、证据(注释者提供独立的事实证据)和权威(注释者作为更广泛人口的代表决定系统输出)。作者认为,这些模型对RLHF流程如何征集、验证和聚合注释有影响。通过回顾RLHF及相关方法的文献,本文展示了这些模型如何被隐含使用,描述了因无意或故意混淆这些模型导致的失败模式,并提供了选择这些模型的规范标准。核心建议是RLHF流程设计者应将注释分解为可分离的维度,并针对每个维度选择最合适的模型,而非寻求单一统一的流程。

英文摘要:

Preference-based alignment methods, most prominently Reinforcement Learning with Human Feedback (RLHF), use the judgments of human annotators to shape large language model behaviour. However, the normative role of these judgments is rarely made explicit. I distinguish three conceptual models of that role. The first is extension: annotators extend the system designers' own judgments about what outputs should be. The second is evidence: annotators provide independent evidence about some facts, whether moral, social or otherwise. The third is authority: annotators have some independent authority (as representatives of the broader population) to determine system outputs. I argue that these models have implications for how RLHF pipelines should solicit, validate and aggregate annotations. I survey landmark papers in the literature on RLHF and related methods to illustrate how they implicitly draw on these models, describe failure modes that come from unintentionally or intentionally conflating them, and offer normative criteria for choosing among them. My central recommendation is that RLHF pipeline designers should decompose annotation into separable dimensions and tailor each pipeline to the model most appropriate for that dimension, rather than seeking a single unified pipeline.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑