推理的动量：策略优化中的密集内在信号
Momentum for Reasoning: Dense Intrinsic Signals in Policy Optimization
AI总结：
针对GRPO在长链推理中因二元奖励导致的零优势崩溃和幻觉确定性失败模式，提出ISPO方法，通过内在信号密集化奖励，在三个基模型和五个数学推理基准上持续优于基线。
摘要：
基于可验证奖励的强化学习已成为激发大型语言模型长链推理的强大范式。然而，现有基于组相对策略优化（GRPO）的方法依赖于二元结果奖励，这引发了两种结构性失败模式：零优势崩溃，即组内所有轨迹共享相同结果导致梯度消失；以及幻觉确定性，即模型在训练后期对错误轨迹变得过度自信。我们通过使用完全从策略自身条件概率计算的内在信号来密集化奖励，解决了这两种模式，并提出了ISPO（内在信号策略优化），它结合了衡量思考轨迹对最终答案信息量的序列级信号，以及令牌级方向性奖励，其幻觉确定性铰链惩罚关键决策令牌上的错误自信预测。在三个基模型和五个数学推理基准上，ISPO持续优于竞争基线，在零优势崩溃最频繁的最难基准上取得最大提升，训练动态诊断证实两种失败模式均被减少。
英文摘要：
Reinforcement learning with verifiable rewards (RLVR) has emerged as a powerful paradigm for eliciting long-chain reasoning in large language models. However, existing methods based on Group Relative Policy Optimization (GRPO) rely on a binary outcome reward, which induces two structural failure modes: Zero-Advantage Collapse, in which all rollouts in a group share the same outcome and the gradient vanishes, and Hallucinated Certainty, in which the model becomes increasingly confident on incorrect rollouts late in training. We address both modes by densifying the reward with intrinsic signals computed entirely from the policy's own conditional probabilities, and propose ISPO (Intrinsic Signal Policy Optimization, which combines a sequence-level signal measuring how informative the thinking trajectory is for the final answer, with a token-level directional reward whose hallucinated-certainty hinge penalizes confidently-wrong predictions at critical decision tokens. Across three base models and five mathematical reasoning benchmarks, ISPO consistently outperforms competitive baselines, with the largest gains on the hardest benchmarks where zero-advantage collapse is most frequent, and training-dynamics diagnostics confirm that both failure modes are decreased.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
