R^3: 为大语言模型强化学习的回放、反思与奖励排名
R^3: Replay, Reflection, and Ranking Rewards for LLM Reinforcement Learning
AI总结:
R^3通过回放、反思和奖励排名机制,提升大语言模型在强化学习中的稳定性和效率,实现数学领域性能突破。
摘要:
大型推理模型(LRMs)旨在通过结构化推理解决多样且复杂的问题。最近基于群体的策略优化方法在无需依赖过程级注释的情况下,展示了实现稳定优势估计的潜力。然而,这些方法依赖于同一批内高质量样本诱导的优势差距,这在挑战性任务中,当组内优势崩溃时会使训练过程变得脆弱且低效。为了解决这些问题,我们提出了一种名为R^3的强化学习机制,从三个方面入手:(1)一种跨上下文回放策略,通过回忆相同查询的历史轨迹中的有价值示例来维持组内优势;(2)一种基于上下文的自我反思机制,使模型能够通过利用过去的失败来优化输出;(3)一种结构熵奖励,通过基于token级熵模式对截断或失败的样本进行排名,分配相对奖励,捕捉局部探索和全局稳定性。我们将在Deepseek-R1-Distill-Qwen-1.5B上实现该方法,并在数学领域训练DeepscaleR-40k。实验表明,我们的方法在多个数学基准上实现了最先进的性能,代表了显著的改进和更少的推理token。代码和模型将被发布。
英文摘要:
Large reasoning models (LRMs) aim to solve diverse and complex problems through structured reasoning. Recent advances in group-based policy optimization methods have shown promise in enabling stable advantage estimation without reliance on process-level annotations. However, these methods rely on advantage gaps induced by high-quality samples within the same batch, which makes the training process fragile and inefficient when intra-group advantages collapse under challenging tasks. To address these problems, we propose a reinforcement learning mechanism named \emph{\textbf{R^3}} that along three directions: (1) a \emph{cross-context \underline{\textbf{R}}eplay} strategy that maintains the intra-group advantage by recalling valuable examples from historical trajectories of the same query, (2) an \emph{in-context self-\underline{\textbf{R}}eflection} mechanism enabling models to refine outputs by leveraging past failures, and (3) a \emph{structural entropy \underline{\textbf{R}}anking reward}, which assigns relative rewards to truncated or failed samples by ranking responses based on token-level entropy patterns, capturing both local exploration and global stability. We implement our method on Deepseek-R1-Distill-Qwen-1.5B and train it on the DeepscaleR-40k in the math domain. Experiments demonstrate our method achieves SoTA performance on several math benchmarks, representing significant improvements and fewer reasoning tokens over the base models. Code and model will be released.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。