从因果视角缓解RLHF中的长度偏差
Mitigating Length Bias in RLHF through a Causal Lens
AI总结：
针对RLHF奖励模型偏好长回复的长度偏差问题，提出因果框架下的反事实数据增强方法训练奖励模型，有效降低长度偏差，提升奖励建模的鲁棒性与内容敏感度。
摘要：
基于人类反馈的强化学习（RLHF）被广泛用于使大语言模型（LLM）与人类偏好对齐。然而，经RLHF训练的奖励模型往往存在长度偏差——即一种将冗长性与质量混为一谈、系统性偏好更长回复的倾向。我们提出了一个因果框架，用于分析和缓解RLHF奖励建模中的长度偏差。我们方法的核心是一种反事实数据增强方法，该方法生成的回复对旨在将内容质量与冗长性分离开。这些反事实示例随后被用于训练奖励模型，使其能够独立于冗长性、基于内容质量评估回复。具体而言，我们构建了两类样本对：（1）内容相似、长度不同的样本对；（2）长度相近、内容不同的样本对。实证评估表明，我们的方法降低了奖励分配中的长度偏差，使策略模型输出更简洁、更聚焦内容。这些发现证明，所提方法可有效降低长度偏差，提升RLHF流程中奖励建模的鲁棒性与内容敏感度。
英文摘要：
Reinforcement learning from human feedback (RLHF) is widely used to align large language models (LLMs) with human preferences. However, RLHF-trained reward models often exhibit length bias -- a systematic tendency to favor longer responses by conflating verbosity with quality. We propose a causal framework for analyzing and mitigating length bias in RLHF reward modeling. Central to our approach is a counterfactual data augmentation method that generates response pairs designed to isolate content quality from verbosity. These counterfactual examples are then used to train the reward model, enabling it to assess responses based on content quality independently of verbosity. Specifically, we construct (1) length-divergent pairs with similar content and (2) content-divergent pairs of similar length. Empirical evaluations show that our method reduces length bias in reward assignment and leads to more concise, content-focused outputs from the policy model. These findings demonstrate that the proposed approach effectively reduces length bias and improves the robustness and content sensitivity of reward modeling in RLHF pipelines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
