道阻且长：探究RLHF中的长度相关性
A Long Way to Go: Investigating Length Correlations in RLHF
AI总结：
该研究揭示RLHF对齐大模型时的奖励提升多由响应长度增加驱动，偏差主要源于鲁棒性不足的奖励模型，且纯长度奖励可复现大部分RLHF下游增益。
摘要：
基于人类反馈的强化学习（Reinforcement Learning from Human Feedback, RLHF）在对齐大语言模型方面已取得巨大成功，公开偏好数据集支撑了更广泛的实验，尤其针对对话、网页问答等任务的“有用性”优化。然而在这些进展之外，RLHF也常导致模型生成更长的输出。本文在三种不同场景下证明，优化响应长度是RLHF效果背后的重要因素，其影响远超此前认知。研究RL优化最大化奖励的策略时，我们发现奖励提升主要由响应长度增加驱动，而非其他特征。实际上，即便仅使用基于长度的奖励，也能复现RLHF相对于监督微调模型的大部分下游效果提升。通过测试一组全面的长度偏差干预方案，我们确定这类偏差的主要来源是奖励模型；对训练动态的研究显示，奖励模型鲁棒性不足，易受偏好数据中长度偏差的影响。
英文摘要：
Great success has been reported using Reinforcement Learning from Human Feedback (RLHF) to align large language models, with open preference datasets enabling wider experimentation, particularly for "helpfulness" in tasks like dialogue and web question answering. Alongside these improvements, however, RLHF also often drives models to produce longer outputs. This paper demonstrates, on three diverse settings, that optimizing for response length is, much more than previously thought, a significant factor behind RLHF. Studying the strategies RL optimization uses to maximize reward, we find improvements in reward to largely be driven by increasing response length, instead of other features. Indeed, we find that even a purely length-based reward reproduces most downstream RLHF improvements over supervised fine-tuned models. Testing a comprehensive set of length-countering interventions, we identify the dominant source of these biases to be reward models, which, by studying training dynamics, we find are non-robust and easily influenced by length biases in preference data.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
