优先过程而非结果:奖励潜在思维轨迹改善循环语言模型的推理能力
Prioritize the Process, Not Just the Outcome: Rewarding Latent Thought Trajectories Improves Reasoning in Looped Language Models
AI总结:
针对循环语言模型(LoopLM)中标准强化学习(如GRPO)仅奖励最终潜在状态导致推理改进失败的问题,提出RLTT框架,通过在整个潜在推理轨迹上分配奖励实现密集的轨迹级信用分配,显著提升数学推理性能并泛化至非数学任务。
摘要:
循环语言模型(LoopLMs)在生成token之前执行多步潜在推理,并在较小的参数预算下在推理基准上优于传统LLM。然而,使用强化学习进一步改进LoopLM推理的尝试失败了——诸如群体相对策略优化(GRPO)等标准目标仅对最终潜在状态分配信用,与模型的内部计算存在根本性不匹配。为解决此问题,我们引入了RLTT(奖励潜在思维轨迹),这是一种强化学习框架,将奖励分布在整个潜在推理轨迹上。RLTT提供密集的、轨迹级别的信用分配,无需依赖外部验证器,并且可以直接替代GRPO,开销可忽略不计。在相同训练和推理条件下,使用Ouro-1.4B/2.6B-Thinking进行的大量实验中,RLTT在具有挑战性的数学推理基准上比GRPO取得了统计上显著的改进,在1.4B规模上,MATH-500、AIME24/26和BeyondAIME的平均准确率提高了+5.8%,在2.6B规模上提高了+10.9%。尽管仅在数学上训练,RLTT也能有效迁移到非数学推理基准,证明了轨迹级信用分配在LoopLMs中强化学习的有效性。代码可在https://github.com/jonwill8/RLTT.git获取。
英文摘要:
Looped Language Models (LoopLMs) perform multi-step latent reasoning prior to token generation and outperform conventional LLMs on reasoning benchmarks at smaller parameter budgets. However, attempts to further improve LoopLM reasoning with reinforcement learning have failed - standard objectives such as Group Relative Policy Optimization (GRPO) only assign credit to the final latent state, creating a fundamental mismatch with the model's internal computation. To resolve this, we introduce RLTT (Reward Latent Thought Trajectories), a reinforcement learning framework which distributes reward across the full latent reasoning trajectory. RLTT provides dense, trajectory-level credit assignment without relying on external verifiers and can directly replace GRPO with negligible overhead. Across extensive experiments with Ouro-1.4B/2.6B-Thinking under identical training and inference conditions, RLTT yields statistically significant improvements over GRPO on challenging mathematical reasoning benchmarks, improving mean accuracy over MATH-500, AIME24/26, and BeyondAIME by +5.8% on the 1.4B scale, and +10.9% on the 2.6B scale. Despite being trained exclusively on mathematics, RLTT also transfers effectively to non-mathematical reasoning benchmarks, demonstrating the effectiveness of trajectory-level credit assignment for reinforcement learning in LoopLMs. Code is available at https://github.com/jonwill8/RLTT.git.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。