Retro-Search：探索未选择路径以实现更深入高效的推理
Retro-Search: Exploring Untaken Paths for Deeper and Efficient Reasoning
AI总结：
针对大型推理模型推理轨迹非最优的问题，提出受MCTS启发的Retro-Search算法，通过回顾修正推理路径蒸馏更优更短轨迹，在自我改进和弱到强改进场景中均实现推理长度缩短与性能提升，反驳了搜索算法无关性观点。
摘要：
大型推理模型通过冗长、详尽的推理轨迹展现出卓越的推理能力。在这些推理轨迹上进行监督微调（也称为蒸馏），是提升学生模型推理能力的一种经济高效的方法。然而，实证观察表明，这些推理轨迹往往并非最优，在不同思路间过度切换，导致思考不足、思考过度甚至退化响应。我们引入Retro-Search——一种受MCTS启发的搜索算法，用于从大型推理模型中蒸馏更高质量的推理路径。Retro-Search通过回顾性修正推理路径，发现更优且更短的轨迹，从而使学生模型在推理能力增强的同时，推理更短、推理速度更快。该方法支持两种应用场景：自我改进（模型在自身经Retro-Search处理的思维轨迹上微调）和弱到强改进（较弱模型通过Retro-Search修正较强模型的思维轨迹）。在自我改进方面，基于自身Retro-Search轨迹微调的R1-distill-7B在七个数学基准上平均推理长度减少31.2%，性能提升7.7%；在弱到强改进方面，使用小20倍的R1-distill-32B作为Retro-Search器修正OpenThoughts数据集中R1-671B的轨迹，基于该精炼数据微调的Qwen2.5-32B性能与R1-distill-32B相当，推理长度减少11.3%，性能提升2.4%（相较于原始OpenThoughts数据微调结果）。我们的工作反驳了近期质疑搜索算法在大型推理模型时代相关性的观点，证明即使对于前沿模型，算法进步仍有机会。
英文摘要：
Large reasoning models exhibit remarkable reasoning capabilities via long, elaborate reasoning trajectories. Supervised fine-tuning on such reasoning traces, also known as distillation, can be a cost-effective way to boost reasoning capabilities of student models. However, empirical observations reveal that these reasoning trajectories are often suboptimal, switching excessively between different lines of thought, resulting in under-thinking, over-thinking, and even degenerate responses. We introduce Retro-Search, an MCTS-inspired search algorithm, for distilling higher quality reasoning paths from large reasoning models. Retro-Search retrospectively revises reasoning paths to discover better, yet shorter traces, which can then lead to student models with enhanced reasoning capabilities with shorter, thus faster inference. Our approach can enable two use cases: self-improvement, where models are fine-tuned on their own Retro-Search-ed thought traces, and weak-to-strong improvement, where a weaker model revises stronger model's thought traces via Retro-Search. For self-improving, R1-distill-7B, fine-tuned on its own Retro-Search-ed traces, reduces the average reasoning length by 31.2% while improving performance by 7.7% across seven math benchmarks. For weak-to-strong improvement, we retrospectively revise R1-671B's traces from the OpenThoughts dataset using R1-distill-32B as the Retro-Search-er, a model 20x smaller. Qwen2.5-32B, fine-tuned on this refined data, achieves performance comparable to R1-distill-32B, yielding an 11.3% reduction in reasoning length and a 2.4% performance improvement compared to fine-tuning on the original OpenThoughts data. Our work counters recently emergent viewpoints that question the relevance of search algorithms in the era of large reasoning models, by demonstrating that there are still opportunities for algorithmic advancements, even for frontier models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
