思维种子：在语言模型中利用历史多样性进行位置感知强化学习
Thinking Seeds: Leveraging Historical Diversity for Position-Aware RL in LLMs
AI总结：
研究语言模型在线强化学习问题，提出思维种子这一令牌级混合策略框架，利用模型历史检查点作离线前缀，通过令牌级重要性比率有效利用历史多样性，实验表明其性能优于标准在线训练和现有离线扩展。
摘要：
语言模型训练后的在线强化学习存在一个根本矛盾：随着训练推进，策略熵崩溃且采样多样性减少，使模型“忘记”早期探索能力。虽离线数据可恢复多样性，但现有方法在序列级别混合整个轨迹，导致严重策略不匹配和训练不稳定。我们认为核心问题不是是否使用离线数据，而是其在序列中的位置。基于此，我们提出思维种子，一种令牌级混合策略框架，用模型自身历史检查点作为离线前缀，为推理提供多样起点，关键延续部分在线生成以保持梯度质量。通过令牌级重要性比率，思维种子有效利用历史多样性且不影响训练稳定性。跨模型和数学推理基准的大量实验表明，思维种子始终优于标准在线训练和现有离线扩展。我们的分析揭示，该方法保持更高有效熵，减少裁剪导致的梯度损失，扩展可探索解空间，阐明位置感知混合策略建模如何改善语言模型强化学习中的探索和最终性能。
英文摘要：
On-policy reinforcement learning (RL) for language model post-training suffers from a fundamental tension: as training progresses, policy entropy collapses and sampling diversity diminishes, causing the model to ``forget'' its own earlier exploratory capacity. While off-policy data can restore diversity, existing methods mix entire trajectories at the sequence level, introducing severe policy mismatch and training instability. We argue that the core question is not \emph{whether} to use off-policy data, but \emph{where} in the sequence it should appear. Based on this insight, we propose \textbf{Thinking Seeds}, a token-level mix-policy framework that uses the model's own historical checkpoints as off-policy prefixes, providing diverse starting points for reasoning, while the critical continuation is generated on-policy to preserve gradient quality. Through token-level importance ratios, Thinking Seeds effectively leverages historical diversity without compromising training stability. Extensive experiments across models and mathematical reasoning benchmarks demonstrate that Thinking Seeds consistently outperforms standard on-policy training and existing off-policy extensions. Our analysis reveals that the method maintains higher effective entropy, reduces gradient loss from clipping, and expands the explorable solution space, clarifying how position-aware mix-policy modeling improves both exploration and final performance in LLM RL.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
