训练LLMs进行分而治之推理以提高测试时的可扩展性
Training LLMs for Divide-and-Conquer Reasoning Elevates Test-Time Scalability
AI总结：
本文提出一种端到端强化学习框架，通过分而治之推理提升LLMs在复杂任务上的推理能力和测试时可扩展性。
摘要：
大型语言模型（LLMs）通过逐步链式思维（CoT）推理展示了强大的推理能力。然而，在模型能力极限时，CoT往往证明不足，其严格顺序的性质限制了测试时的可扩展性。一个潜在的替代方法是分而治之（DAC）推理，它将复杂问题分解为子问题以促进更有效的解决方案探索。尽管有前景，我们的分析揭示了通用目的后训练与DAC风格推理由本不一致，这限制了模型完全利用这一潜力的能力。为了弥合这一差距并充分释放LLMs在最具挑战性任务上的推理能力，我们提出一个端到端的强化学习（RL）框架来增强其DAC风格推理能力。在每一步，策略将问题分解为一组子问题，按顺序解决它们，并根据子问题解决方案解决原始问题，分解和解决都整合到RL训练中。在同等训练下，我们的DAC风格框架使模型具有更高的性能上限和更强的测试时可扩展性，在竞争级基准上，比CoT在Pass@1上高出8.6%，在Pass@32上高出6.3%。
英文摘要：
Large language models (LLMs) have demonstrated strong reasoning capabilities through step-by-step chain-of-thought (CoT) reasoning. Nevertheless, at the limits of model capability, CoT often proves insufficient, and its strictly sequential nature constrains test-time scalability. A potential alternative is divide-and-conquer (DAC) reasoning, which decomposes a complex problem into subproblems to facilitate more effective exploration of the solution. Although promising, our analysis reveals a fundamental misalignment between general-purpose post-training and DAC-style inference, which limits the model's capacity to fully leverage this potential. To bridge this gap and fully unlock LLMs' reasoning capabilities on the most challenging tasks, we propose an end-to-end reinforcement learning (RL) framework to enhance their DAC-style reasoning capacity. At each step, the policy decomposes a problem into a group of subproblems, solves them sequentially, and addresses the original one conditioned on the subproblem solutions, with both decomposition and solution integrated into RL training. Under comparable training, our DAC-style framework endows the model with a higher performance ceiling and stronger test-time scalability, surpassing CoT by 8.6% in Pass@1 and 6.3% in Pass@32 on competition-level benchmarks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
