$λ$-GRPO：通过可学习令牌偏好统一GRPO框架
$λ$-GRPO: Unifying the GRPO Frameworks with Learnable Token Preferences
AI总结：
本文提出$λ$-GRPO，通过引入可学习参数统一GRPO框架，自适应调整令牌级权重以缓解长度偏差，在数学推理基准上较GRPO和DAPO持续提升准确率，且无需额外计算成本。
摘要：
基于人类反馈的强化学习（RLHF）一直是提升大型语言模型（LLMs）推理能力的主要方法。近年来，基于可验证奖励的强化学习（RLVR）通过用基于规则的验证器取代奖励模型和价值模型，简化了这一范式。一个突出的例子是组相对策略优化（GRPO）。然而，GRPO固有地存在长度偏差，因为相同的优势被均匀地分配给响应中的所有令牌。因此，较长的响应将奖励分散到更多令牌上，从而对梯度更新产生不成比例的贡献。DAPO和Dr. GRPO等几种变体修改了损失的令牌级聚合，但这些方法仍是启发式的，并且对其隐含的令牌偏好的可解释性有限。在这项工作中，我们探索了允许模型在优化过程中学习自身令牌偏好的可能性。我们将现有框架统一到一个单一公式下，并引入一个可学习参数$λ$，自适应地控制令牌级权重。我们用$λ$-GRPO表示我们的方法，并发现$λ$-GRPO在多个数学推理基准上相对于原始GRPO和DAPO取得了持续改进。在参数规模为1.5B、3B和7B的Qwen2.5模型上，与GRPO相比，$λ$-GRPO分别将平均准确率提高了$+1.9\%$、$+1.0\%$和$+1.7\%$。重要的是，这些提升无需修改训练数据或增加额外计算成本，突显了学习令牌偏好的有效性和实用性。
英文摘要：
Reinforcement Learning with Human Feedback (RLHF) has been the dominant approach for improving the reasoning capabilities of Large Language Models (LLMs). Recently, Reinforcement Learning with Verifiable Rewards (RLVR) has simplified this paradigm by replacing the reward and value models with rule-based verifiers. A prominent example is Group Relative Policy Optimization (GRPO). However, GRPO inherently suffers from a length bias, since the same advantage is uniformly assigned to all tokens of a response. As a result, longer responses distribute the reward over more tokens and thus contribute disproportionately to gradient updates. Several variants, such as DAPO and Dr. GRPO, modify the token-level aggregation of the loss, yet these methods remain heuristic and offer limited interpretability regarding their implicit token preferences. In this work, we explore the possibility of allowing the model to learn its own token preference during optimization. We unify existing frameworks under a single formulation and introduce a learnable parameter $λ$ that adaptively controls token-level weighting. We use $λ$-GRPO to denote our method, and we find that $λ$-GRPO achieves consistent improvements over vanilla GRPO and DAPO on multiple mathematical reasoning benchmarks. On Qwen2.5 models with 1.5B, 3B, and 7B parameters, $λ$-GRPO improves average accuracy by $+1.9\%$, $+1.0\%$, and $+1.7\%$ compared to GRPO, respectively. Importantly, these gains come without any modifications to the training data or additional computational cost, highlighting the effectiveness and practicality of learning token preferences.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
