应对长度膨胀而不产生权衡：用于强化学习的群体相对奖励重缩放
Tackling Length Inflation Without Trade-offs: Group Relative Reward Rescaling for Reinforcement Learning
AI总结：
GR$^3$通过乘法重缩放范式解决强化学习中的长度膨胀问题，实现通用、连续且奖励依赖的门控机制，有效减少冗余推理并提升训练性能。
摘要：
强化学习显著增强了大语言模型的能力，但面临一个关键问题：长度膨胀，即模型采用冗长或低效的推理方式以最大化奖励。先前的方法难以在一般性和无损失地解决这一挑战，主要因为加法惩罚引入了补偿效应，导致优化捷径，而启发式门控策略在二元反馈之外缺乏通用性。为弥合这一差距，我们提出了群体相对奖励重缩放（GR$^3$），将长度控制重新定义为乘法重缩放范式，有效建立了通用的、连续的、依赖奖励的门控机制。为进一步确保无损失优化，我们引入了群体相对正则化和优势意识校准，这些动态适应实例难度并保留高质量轨迹的优势信号。实证表明，在RLHF和RLVR设置中，GR$^3$保持了与标准GRPO相当的训练动态和下游性能，同时显著减轻了长度膨胀，优于最先进的长度正则化基线。
英文摘要：
Reinforcement learning significantly enhances LLM capabilities but suffers from a critical issue: length inflation, where models adopt verbosity or inefficient reasoning to maximize rewards. Prior approaches struggle to address this challenge in a general and lossless manner, primarily because additive penalties introduce a compensatory effect that creates optimization shortcuts, while heuristic gating strategies lack generality beyond binary feedback. To bridge this gap, we present Group Relative Reward Rescaling (GR$^3$), which reframes length control as a multiplicative rescaling paradigm, effectively establishing a generalized, continuous, and reward-dependent gating mechanism. To further ensure lossless optimization, we incorporate group-relative regularization and advantage-aware calibration, which dynamically adapt length budgets to instance difficulty and preserve the advantage signal of high-quality trajectories. Empirically, across both RLHF and RLVR settings, GR$^3$~maintains training dynamics and downstream performance comparable to standard GRPO while significantly mitigating length inflation, outperforming state-of-the-art length-regularized baselines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
