揭秘隐状态循环：基于在线强化学习的可切换潜在推理
Demystifying Hidden-State Recurrence: Switchable Latent Reasoning with On-Policy Reinforcement Learning
AI总结：
提出SWITCH框架，通过离散边界令牌使隐状态循环推理兼容在线强化学习，并支持因果机制分析，实验表明其优于现有方法。
摘要：
潜在思维链通过用连续的隐状态循环替换可见推理轨迹来压缩推理，但现有公式难以用标准在线强化学习（RL）优化，且难以进行因果解释。我们的关键见解是，一对显式的边界令牌可以同时解决这两个问题：离散的进入和退出锚点使潜在块与标准在线RL兼容，并且相同的锚点为机制分析提供了自然立足点。基于此，我们提出SWITCH，一个可切换的潜在推理框架。模型发出<swi>进入潜在模式，</swi>退出。由于边界是普通的离散令牌，GRPO策略比率在每个决策点都有明确定义。相同的锚点也使潜在步骤暴露于直接探测和因果干预。我们通过可见到潜在的课程和Switch-GRPO目标训练模型，该目标通过循环潜在计算传播梯度。SWITCH在相似规模下始终优于先前的隐状态循环潜在推理方法。通过边界令牌的机制分析进一步揭示了三个发现：（i）<swi>是一个尖锐局部化的学习切换策略，而非风格化伪影；（ii）它开启的潜在步骤执行特定于问题的、因果重要的计算，而非作为惰性占位符；（iii）该计算集中在进入时的单个隐状态转换上。这些结果表明，隐状态循环潜在推理既可RL训练，又可进行直接机制分析，包括在线RL本身如何从内部改进模型。
英文摘要：
Latent chain-of-thought compresses reasoning by replacing visible reasoning traces with continuous hidden-state recurrence, but existing formulations are difficult to optimize with standard on-policy reinforcement learning (RL) and hard to interpret causally. Our key insight is that a single pair of explicit boundary tokens can address both issues at once: discrete entry and exit anchors make the latent block compatible with standard on-policy RL, and the same anchors offer a natural foothold for mechanistic analysis. Motivated by this, we propose SWITCH, a switchable latent reasoning framework. The model emits <swi> to enter latent mode and </swi> to exit. Because the boundaries are ordinary discrete tokens, the GRPO policy ratio is well-defined at every decision point. The same anchors also expose the latent steps to direct probing and causal intervention. We train the model with a visible-to-latent curriculum and a Switch-GRPO objective that propagates gradients through recurrent latent computation. SWITCH consistently outperforms prior hidden-state-recurrence latent reasoning approaches at similar scale. Mechanistic analysis through the boundary tokens further reveals three findings: (i) <swi> is a sharply localised, learned switching policy rather than a stylistic artefact; (ii) the latent step it opens performs problem-specific, causally important computation rather than acting as an inert placeholder; and (iii) that computation is concentrated at a single hidden-state transition on entry. Together, these results show that hidden-state-recurrence latent reasoning is both RL-trainable and open to direct mechanistic analysis, including of how on-policy RL itself improves the model from the inside.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
