arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2606.04889cs.CL
收藏

GRAIL: 基于梯度重加权优势的可验证奖励强化学习

GRAIL: Gradient-Reweighted Advantages for Reinforcement Learning with Verifiable Rewards

DeCLaRe Lab, Nanyang Technological University(DeCLaRe实验室,南洋理工大学)

Tej Deep Pala, Vernon Toh, Soujanya Poria

AI总结:

针对强化学习中统一优势分配导致梯度信号稀释的问题,提出基于梯度激活显著性的令牌级优势重加权方法GRAIL,无需过程级监督即可提升推理对齐,在多个模型上平均准确率提升3.60%。

摘要:

基于可验证奖励的强化学习(如GRPO)现在已成为提升大语言模型(LLMs)数学推理能力的常见方法。然而,当前方法通常将单个序列级优势广播到所有令牌,或使用昂贵的过程奖励模型(PRMs)进行步骤级监督。统一优势分配假设所有令牌对最终奖励的贡献相同。这会稀释梯度信号,因为存在缺陷的推理步骤和填充词与有效的逻辑推理得到同等强度的更新。为解决此问题,我们引入了梯度重加权优势(GRAIL),一种内在的令牌级优势重加权方法。GRAIL使用梯度激活显著性,将更多权重赋予那些对最终答案局部更敏感的令牌。在来自Qwen3、R1-distilled和OctoThinker家族的五个模型上的评估表明,GRAIL始终优于GRPO。GRAIL在准确率上平均提升3.60%,在Pass@3上平均提升3.05%,表明无需过程级监督即可实现细粒度的推理对齐。

英文摘要:

Reinforcement learning with verifiable rewards (e.g. GRPO) is now a common way to improve mathematical reasoning in Large Language Models (LLMs). However, current methods usually broadcast one sequence-level advantage to all tokens, or use costly process reward models (PRMs) for step-level supervision. Uniform advantage distribution assumes that all tokens contribute equally to the final reward. This dilutes the gradient signal, since flawed reasoning steps and filler words are updated as strongly as valid logical inferences. To address this, we introduce Gradient-Reweighted Advantage (GRAIL), an intrinsic token-wise advantage reweighting method. GRAIL uses gradient-activation saliency to place more weight on tokens that are more locally sensitive to the final answer. Evaluations across five models from the Qwen3, R1-distilled and OctoThinker families show that GRAIL consistently outperforms GRPO. GRAIL achieved an average improvement of 3.60% in accuracy and 3.05% in Pass@3, demonstrating that fine-grained reasoning alignment can be achieved without process-level supervision.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑