arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.01002cs.AI
收藏

如何通过RLHF放大阿谀行为

How RLHF Amplifies Sycophancy

harvard(哈佛大学)

Itai Shapira, Gerdus Benade, Ariel D. Procaccia

AI总结:

本文研究了RLHF如何通过放大机制增强阿谀行为,提出了一种训练干预措施以减少这种偏差。

摘要:

大型语言模型在基于偏好的后训练过程中往往表现出增加的阿谀行为,即使这种行为与事实准确性或理性判断相冲突,也会更倾向于肯定用户的陈述或隐含信念。我们通过正式分析人类反馈对齐如何增加这种失败模式,识别出一个显式的放大机制,该机制因果地将优化与学习到的奖励联系到用于对齐的人类偏好数据中的偏差。我们显示,行为漂移的方向由基础策略下内核信念信号与学习到的奖励之间的协方差决定,且一阶效应简化为一个简单的均值差距条件。随后,我们分析了在类似布拉德利-泰利的随机效用模型下,从成对比较中学习奖励,并确定何时人类标注者偏见会引发这种奖励差距。接着,我们提出了一种训练时的干预措施,旨在中和放大机制本身。在所有防止阿谀行为增加的后训练策略中,我们确定了最接近无约束后训练策略的唯一策略,并推导出相应的最小奖励修正作为闭合形式的一致性惩罚。计算实验发现,奖励差距普遍存在,并在所有考虑的配置中导致行为漂移。

英文摘要:

Large language models often exhibit increased sycophantic behavior after preference-based post-training, showing a stronger tendency to affirm a user's stated or implied belief even when this conflicts with factual accuracy or sound judgment. We present a formal analysis of how alignment from human feedback can increase this failure mode by identifying an explicit amplification mechanism that causally links optimization against a learned reward to bias in the human preference data used for alignment. We show that the direction of behavioral drift is determined by a covariance under the base policy between endorsing the belief signal in the prompt and the learned reward, and that the first-order effect reduces to a simple mean-gap condition. We then analyze reward learning from pairwise comparisons under random utility models like Bradley-Terry and characterize when bias in human annotators' preferences induces this reward gap. Next, we propose a training-time intervention designed to neutralize the amplification mechanism itself. Among all post-trained policies that prevent sycophantic behavior from increasing, we characterize the unique policy closest in KL divergence to the unconstrained post-trained policy, and derive the corresponding minimal reward correction as a closed-form agreement penalty. Computational experiments find that reward gaps are common and cause behavioral drift in all the configurations considered.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑