arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.00759cs.CLcs.AI
收藏

适应性能力分解用于解锁大型推理模型有效强化学习

Adaptive Ability Decomposing for Unlocking Large Reasoning Model Effective Reinforcement Learning

Renmin University of China(中国人民大学)

Zhipeng Chen, Xiaobo Qin, Wayne Xin Zhao, Youbin Wu, Ji-Rong Wen

AI总结:

本文提出A$^2$D方法,通过分解复杂问题提升RLVR效果,适用于多种强化学习算法。

摘要:

基于可验证奖励的强化学习(RLVR)已显示出增强大型语言模型(LLMs)推理能力的巨大潜力。然而,由于在RLVR过程中提供的信息有限,模型只能进行大量盲目的探索,这往往导致在具有挑战性的问题上失败。为了在不依赖教师模型的情况下为RLVR过程提供额外信息,我们提出了A$^2$D,一种适应性能力分解方法,用于增强RLVR的有效性。具体而言,我们首先通过RLVR训练一个分解器,无需蒸馏,使其能够将复杂问题分解成一组更简单的子问题。接下来,我们使用这个分解器为训练数据集中的每个问题注释子问题,然后在具有子问题指导的RLVR下训练推理器。为了更好地理解A$^2$D,我们首先将其性能与竞争性基线进行比较,显示其有效性。接下来,我们观察到我们的方法作为一个即插即用的模块,可以应用于不同的RLVR算法。此外,我们对分解器进行了分析,揭示了RLVR过程如何影响其性能和行为,以及哪种类型的指导更适合增强推理器的探索和利用能力。

英文摘要:

Reinforcement learning with verifiable rewards (RLVR) has shown great potential to enhance the reasoning ability of large language models (LLMs). However, due to the limited amount of information provided during the RLVR process, the model can only engage in largely blind exploration, which often results in failure on challenging problems. To provide additional information for the RLVR process without relying on a teacher model, we propose A$^2$D, an Adaptive Ability Decomposing method for enhancing the effectiveness of RLVR. Specifically, we first train a decomposer via RLVR without distillation, enabling it to decompose complex questions into a set of simpler sub-questions. Next, we use this decomposer to annotate sub-questions for each question in the training dataset, and then train the reasoner under RLVR with sub-question guidance. To better understand A$^2$D, we first compare its performance with competitive baselines, showing its effectiveness. Next, we observe that our method functions as a plug-and-play module that can be applied to different RLVR algorithms. Furthermore, we conduct an analysis of the decomposer, revealing how the RLVR process affects its performance and behavior, and which type of guidance is better suited for enhancing the reasoner's exploration and exploitation abilities.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑