arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2601.23027cs.LG
收藏

分而治之的CoT:通过并行推理减少延迟的强化学习

Divide-and-Conquer CoT: RL for Reducing Latency via Parallel Reasoning

Stanford University(斯坦福大学)

Arvind Mahankali, Kaiyue Wen, Tengyu Ma

AI总结:

通过并行推理和强化学习,DC-CoT在减少延迟的同时保持高精度,适用于数学推理任务。

摘要:

长链式推理(Long CoT)现在是最先进的LLM的核心,特别是在数学推理中。然而,LLM生成过程高度序列化,长CoT导致高延迟。我们提出训练分而治之的CoT(DC-CoT)以减少延迟。通过DC-CoT,模型可以充当导演,识别可以在推理过程中并行执行的子任务,然后派发工人执行子任务。我们的目标是实现高精度,同时具有低最长路径长度,这是响应所需延迟的理论度量。我们从一个长CoT基础模型(DeepScaleR-1.5B-Preview)开始,首先使用小规模定制演示集进行SFT来初始化其派发工人的能力。由于SFT显著降低精度,我们设计了多阶段强化学习算法,结合各种数据过滤策略,以在减少最长路径长度的同时恢复精度。在多个基准测试中,包括AIME 2024和HMMT 2025,DC-CoT在精度上与DeepScaleR-1.5B-Preview相似,同时将最长路径长度减少了35-40%。我们的代码、SFT数据集和模型可在https://github.com/amahankali10/DC_CoT_RL_for_Low_Latency_CoT_with_Parallel_Reasoning上公开获取。

英文摘要:

Long chain-of-thought reasoning (Long CoT) is now fundamental to state-of-the-art LLMs, especially in mathematical reasoning. However, LLM generation is highly sequential, and long CoTs lead to a high latency. We propose to train Divide-and-Conquer CoT (DC-CoT) to reduce the latency. With DC-CoT, the model can act as a director that identifies distinct subtasks that can be performed in parallel in its reasoning process, and then spawns workers to execute the subtasks. Our goal is to achieve high accuracy, with a low longest path length, which is a theoretical measure of the latency needed for the response. We start with a long CoT base model (DeepScaleR-1.5B-Preview), and first use SFT with a small curated demonstration set to initialize its ability to spawn workers in a certain format. Because SFT degrades the accuracy significantly, we design a multi-stage RL algorithm, with various data filtering strategies, to recover the accuracy while decreasing the longest path length. Across several benchmarks including AIME 2024 and HMMT 2025, DC-CoT achieves similar accuracy as DeepScaleR-1.5B-Preview while decreasing longest path length by 35-40%. Our code, SFT dataset and models are publicly available at https://github.com/amahankali10/DC_CoT_RL_for_Low_Latency_CoT_with_Parallel_Reasoning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑