高效推理探索：基于状态条件潜在引导与进度指导

中文摘要 AI 辅助

Best-of-$N$是一种广泛使用的复杂推理推理策略，其有效性取决于采样候选是否能够覆盖多样且高质量的推理路径。然而，后训练推理模型常常遭受“探索崩溃”问题，即独立采样多次重复遵循相似的推理路径，限制了增加采样预算带来的收益。现有方法通过促进更广泛的探索来缓解这一问题，但并未明确引导探索朝向能够取得有意义进展的延续方向，导致探索效率有限。为解决这一问题，我们提出了SPS（状态条件进度引导转向），一种无需训练的潜在引导框架。具体而言，SPS构建了一个状态条件方向库，其中包含针对不同前缀状态区域的多个进度引导转向向量。在线推理过程中，SPS根据当前前缀状态检索合适的转向向量，并将其应用于高不确定性转移处，以引导下一步推理朝向有意义的进展。跨多个模型规模和基准的广泛实验表明，SPS始终优于强基线。进一步的分析验证了其关键设计的有效性，并为未来研究提供了有价值的见解。代码可在以下网址获取：https URL。

英文摘要

Best-of-$N$ is a widely used inference strategy for complex reasoning, whose effectiveness depends on whether sampled candidates can cover diverse and high-quality reasoning paths. However, post-trained reasoning models often suffer from \emph{exploration collapse}, where independent rollouts repeatedly follow similar reasoning paths and limit the gains from increasing the rollout budget. Existing methods alleviate this issue by promoting broader exploration, but do not explicitly guide exploration toward continuations that make meaningful progress, resulting in limited exploration efficiency. To address this, we propose \emph{\underline{S}tate-conditioned \underline{P}rogress-guided \underline{S}teering} (SPS), a training-free latent steering framework. Specifically, SPS constructs a state-conditioned Direction Bank containing multiple progress-guided steering vectors for different prefix-state regions. During online inference, SPS retrieves a suitable steering vector based on the current prefix state and applies it at high-uncertainty transitions to guide the next reasoning step toward meaningful progress. Extensive experiments across multiple model scales and benchmarks demonstrate that SPS consistently outperforms strong baselines. Further analyses validate the effectiveness of its key designs and offer valuable insights for future research. The code is available at https://github.com/rattlesnakey/SPS.

发表机构

• The University of Hong Kong（香港大学）
• University of California, Los Angeles（加利福尼亚大学洛杉矶分校）
• Arizona State University（亚利桑那州立大学）
• Simon Fraser University（西蒙弗雷泽大学）
• The University of Manchester（曼彻斯特大学）
• Tencent（腾讯）

机构由 AI 辅助整理，请以论文原文为准。