CPL:关键计划步骤学习提升大语言模型在推理任务中的泛化能力
CPL: Critical Plan Step Learning Boosts LLM Generalization in Reasoning Tasks
AI总结:
本文提出CPL方法,通过在高层计划空间中使用MCTS搜索多样化步骤,并结合Step-APO学习关键计划步骤,仅用GSM8K和MATH训练即显著提升模型在多个域内外推理基准上的泛化性能。
摘要:
后训练,尤其是利用自博弈生成数据进行强化学习(RL),已成为大语言模型(LLMs)的一种新学习范式。然而,扩展强化学习以培养通用推理器仍是一个研究挑战,因为现有方法侧重于任务特定的推理,未能充分解决在更广泛任务范围内的泛化问题。此外,与动作空间有限的传统强化学习不同,大语言模型在无限空间中运行,因此搜索有价值且多样化的策略以有效解决问题至关重要。为解决这一问题,我们提出在高层抽象计划的动作空间内进行搜索,以增强模型泛化能力,并引入关键计划步骤学习(CPL),包括:1)在计划上进行搜索,利用蒙特卡洛树搜索(MCTS)在多步推理任务中探索多样化的计划步骤;2)通过步骤级优势偏好优化(Step-APO)学习关键计划步骤,该方法将通过MCTS获得的步骤偏好优势估计整合到直接偏好优化(DPO)中。这种结合帮助模型有效学习关键计划步骤,同时提升推理能力和泛化能力。实验结果表明,我们的方法仅使用GSM8K和MATH进行训练,不仅显著提升了GSM8K(+10.5%)和MATH(+6.5%)上的性能,还增强了域外推理基准的表现,如HumanEval(+12.2%)、GPQA(+8.6%)、ARC-C(+4.0%)、MMLU-STEM(+2.2%)和BBH(+1.8%)。
英文摘要:
Post-training, particularly reinforcement learning (RL) using self-play-generated data, has become a new learning paradigm for large language models (LLMs). However, scaling RL to develop a general reasoner remains a research challenge, as existing methods focus on task-specific reasoning without adequately addressing generalization across a broader range of tasks. Moreover, unlike traditional RL with limited action space, LLMs operate in an infinite space, making it crucial to search for valuable and diverse strategies to solve problems effectively. To address this, we propose searching within the action space on high-level abstract plans to enhance model generalization and introduce Critical Plan Step Learning (CPL), comprising: 1) searching on plan, using Monte Carlo Tree Search (MCTS) to explore diverse plan steps in multi-step reasoning tasks, and 2) learning critical plan steps through Step-level Advantage Preference Optimization (Step-APO), which integrates advantage estimates for step preference obtained via MCTS into Direct Preference Optimization (DPO). This combination helps the model effectively learn critical plan steps, enhancing both reasoning capabilities and generalization. Experimental results demonstrate that our method, trained exclusively on GSM8K and MATH, not only significantly improves performance on GSM8K (+10.5%) and MATH (+6.5%), but also enhances out-of-domain reasoning benchmarks, such as HumanEval (+12.2%), GPQA (+8.6%), ARC-C (+4.0%), MMLU-STEM (+2.2%), and BBH (+1.8%).
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。