大型推理模型中的层次化思维建模
Modeling Hierarchical Thinking in Large Reasoning Models
AI总结：
本文提出将大型推理模型（LRM）的层次化推理动态近似为有限状态机（FSM）中的轨迹，并通过Q值引导的推理时控制方法实现高效推理优化。
摘要：
大型推理模型（LRM）通过生成长链思维（CoT）序列来解决复杂任务；然而，控制推理轨迹的涌现动态尚未被充分理解，可能导致不一致性和推理病态。在这项工作中，我们提出将LRM的涌现层次化推理动态近似为有限状态机（FSM）中的轨迹，该状态机在六个抽象认知状态之间转换。我们证明这些状态和转换可以在模型的潜在状态中捕获。我们相信这种表示在LRM模型的可解释性和优化中具有不同的应用。例如，通过分析这些转换的拓扑结构，我们识别出推理策略中的统计变化，有助于从失败的推理链中识别出有效的推理链。为了说明这些潜在优势，我们提出了Q值引导转向，一种无需训练的推理时控制方法，将推理视为规划问题。我们估计状态转换的长期效用，并在句子边界处应用稀疏、正交的激活转向，使CoT生成与最优推理策略对齐。使用三个最先进的开源推理模型在四个基准测试（AIME25、MATH-500、GSM8k和GPQA Diamond）上的实验表明，Q值转向策略以“外科手术式”的效率实现了显著的性能提升，通常需要的干预次数比贪婪和加权基线少25倍，这表明通过引导高层认知动态而非微观管理令牌生成，可以有效地控制推理。代码可在 https://github.com/shahariar-shibli/CoT-FSM 获取。
英文摘要：
Large Reasoning Models (LRMs) solve complex tasks by generating long Chain-of-Thought (CoT) sequences; however, the emergent dynamics governing reasoning trajectories are not well understood and can lead to inconsistencies and reasoning pathologies. In this work, we propose to approximate LRM's emerging hierarchical reasoning dynamics as a trajectory within a Finite State Machine (FSM) transitioning among six abstract cognitive states. We demonstrate that these states and transitions can be captured in the latent state of the model. We believe that this representation can have different applications in the interpretability and optimization of LRM models. For example, by analyzing the topology of these transitions, we identify statistical shifts in reasoning strategies that help identify effective reasoning chains from those that fail. To illustrate these potential advantages, we propose Q-Value guided steering, a training-free inference-time control method that treats reasoning as a planning problem. We estimate the long-horizon utility of state transitions and apply sparse, orthogonal activation steering at sentence boundaries to align the CoT generation with optimal reasoning policies. Experiments across four benchmarks (AIME25, MATH-500, GSM8k, and GPQA Diamond) using three state-of-the-art open reasoning models demonstrate that Q-Value steering policy achieves significant performance gains with "surgical" efficiency, often requiring 25 times fewer interventions than greedy and weighted baselines, which suggests that reasoning can be effectively controlled by guiding high-level cognitive dynamics rather than micro-managing token generation. Code is available at: https://github.com/shahariar-shibli/CoT-FSM.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
