CausalRM：基于观察性用户反馈的因果理论奖励建模用于RLHF
CausalRM: Causal-Theoretic Reward Modeling for RLHF from Observational User Feedbacks
AI总结：
本文提出CausalRM框架，通过因果理论处理观察性用户反馈中的噪声和偏见问题，提升RLHF任务性能，实验显示在WildGuardMix和HarmBench上分别提升49.2%和32.7%。
摘要：
尽管强化学习从人类反馈（RLHF）在对齐语言模型方面取得了成功，但当前奖励建模严重依赖于由人类标注者在受控且成本高昂的条件下收集的实验反馈数据。在本文中，我们引入了观察性奖励建模——通过观察性用户反馈（例如点击、复制和点赞）学习奖励模型——作为一种可扩展且成本效益更高的替代方案。我们识别出在此设置中的两个基本挑战：（1）观察性反馈因标注错误而嘈杂，使其偏离真实用户偏好；（2）观察性反馈受用户偏好影响，用户倾向于对感受强烈的响应提供反馈，这导致训练数据与推理数据之间存在分布偏移。为了解决这些挑战，我们提出了CausalRM，一种因果理论奖励建模框架，旨在从观察性反馈中学习无偏的奖励模型。为解决挑战（1），CausalRM引入了一个噪声感知的替代损失项，该项在无噪声条件下与原始损失证明等价，通过显式建模标注错误生成过程。为解决挑战（2），CausalRM使用倾向分数——用户对给定响应提供反馈的概率——来重新加权训练样本，得到一个消除用户偏好偏见的损失函数。在多样化的LLM后端和基准数据集上进行的广泛实验验证了CausalRM能够从嘈杂和偏见的观察性反馈中有效学习准确的奖励信号，并在下游RLHF任务中带来显著的性能提升——包括在WildGuardMix上提升49.2%和在HarmBench上提升32.7%。代码可在我们的项目网站上获取。
英文摘要：
Despite the success of reinforcement learning from human feedback (RLHF) in aligning language models, current reward modeling heavily relies on experimental feedback data collected from human annotators under controlled and costly conditions. In this work, we introduce observational reward modeling -- learning reward models with observational user feedback (e.g., clicks, copies, and upvotes) -- as a scalable and cost-effective alternative. We identify two fundamental challenges in this setting: (1) observational feedback is noisy due to annotation errors, which deviates it from true user preference; (2) observational feedback is biased by user preference, where users preferentially provide feedback on responses they feel strongly about, which creats a distribution shift between training and inference data. To address these challenges, we propose CausalRM, a causal-theoretic reward modeling framework that aims to learn unbiased reward models from observational feedback. To tackle challenge (1), CausalRM introduces a noise-aware surrogate loss term that is provably equivalent to the primal loss under noise-free conditions by explicitly modeling the annotation error generation process. To tackle challenge (2), CausalRM uses propensity scores -- the probability of a user providing feedback for a given response -- to reweight training samples, yielding a loss function that eliminates user preference bias. Extensive experiments across diverse LLM backbones and benchmark datasets validate that CausalRM effectively learns accurate reward signals from noisy and biased observational feedback and delivers substantial performance improvements on downstream RLHF tasks -- including a 49.2% gain on WildGuardMix and a 32.7% improvement on HarmBench. Code is available on our project website.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
