Marco-o1 v2：致力于拓宽推理模型的蒸馏瓶颈
Marco-o1 v2: Towards Widening The Distillation Bottleneck for Reasoning Models
AI总结：
针对小模型蒸馏长思维链数据时面临的学习困难和过度思考偏见，本文提出利用MCTS构建树状CoT数据，并结合思维长度平衡、细粒度DPO等CoT感知方法增强SFT与RL，有效减少幻觉并显著提升模型推理性能。
摘要：
大型推理模型（LRMs），如 OpenAI o1 和 DeepSeek-R1，通过扩展测试时计算并生成长思维链（CoT），展现出了卓越的推理能力。蒸馏——即在 LRMs 生成的数据上进行后训练——是增强较小模型推理能力的一种直接且有效的方法，但面临一个关键瓶颈：我们发现，蒸馏出的长 CoT 数据给小模型带来了学习困难，并在使用监督微调（SFT）和强化学习（RL）方法时导致偏见的继承（即过度思考）。为缓解此瓶颈，我们提出通过蒙特卡洛树搜索（MCTS）从零开始构建基于树的 CoT 数据。随后，我们利用一组 CoT 感知方法，包括思维长度平衡、细粒度 DPO 和联合后训练目标，以在构建的数据上增强 SFT 和 RL。我们在多个基准上进行评估，如数学（GSM8K, MATH, AIME）、指令遵循（Multi-IF）和规划（Blocksworld）。结果表明，通过减少长时间思考中的幻觉，与标准蒸馏模型相比，我们的方法显著提升了蒸馏模型的推理性能。项目主页为 https://github.com/AIDC-AI/Marco-o1。
英文摘要：
Large Reasoning Models(LRMs) such as OpenAI o1 and DeepSeek-R1 have shown remarkable reasoning capabilities by scaling test-time compute and generating long Chain-of-Thought(CoT). Distillation--post-training on LRMs-generated data--is a straightforward yet effective method to enhance the reasoning abilities of smaller models, but faces a critical bottleneck: we found that distilled long CoT data poses learning difficulty for small models and leads to the inheritance of biases (i.e. over-thinking) when using Supervised Fine-tuning (SFT) and Reinforcement Learning (RL) methods. To alleviate this bottleneck, we propose constructing tree-based CoT data from scratch via Monte Carlo Tree Search(MCTS). We then exploit a set of CoT-aware approaches, including Thoughts Length Balance, Fine-grained DPO, and Joint Post-training Objective, to enhance SFT and RL on the constructed data. We conduct evaluation on various benchmarks such as math (GSM8K, MATH, AIME). instruction-following (Multi-IF) and planning (Blocksworld), results demonstrate our approaches substantially improve the reasoning performance of distilled models compared to standard distilled models via reducing the hallucinations in long-time thinking. The project homepage is https://github.com/AIDC-AI/Marco-o1.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
