arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2504.00829cs.CL
收藏

难度感知的分阶段强化学习如何增强LLM的推理能力:一项初步实验研究

How Difficulty-Aware Staged Reinforcement Learning Enhances LLMs' Reasoning Capabilities: A Preliminary Experimental Study

a-m-team(a-m团队)

Yunjie Ji, Sitong Zhao, Xiaoyu Tian, Haotian Wang, Shuaiting Chen, Yiping Peng, Han Zhao, Xiangang Li

AI总结:

本文提出难度感知的分阶段强化学习策略,通过按难度选择训练数据并逐步增加任务难度,有效提升了LLM的推理能力,使1.5B模型在AIME-2024和MATH-500基准上分别达到42.3%和89.5%的准确率。

摘要:

以高效和可扩展的方式增强大型语言模型(LLM)的推理能力,仍然是人工智能研究中的一项根本挑战。本文对难度感知的分阶段强化学习(RL)策略如何大幅提升LLM的推理性能进行了严谨的实验研究。通过系统分析,我们证明,根据明确的难度级别策略性地选择训练数据,可以显著增强RL优化。此外,我们引入了一种分阶段训练方法,通过逐步让模型接触难度不断增加的任务,进一步放大了推理能力。我们的发现表明,在数学推理和代码生成任务上同时训练模型时,会产生显著的跨领域收益。值得注意的是,我们提出的方法使一个1.5B参数的模型在AIME-2024基准上达到了42.3%的准确率,在MATH-500基准上达到了89.5%的准确率。这些结果突显了我们的方法在提升LLM推理水平方面的有效性。我们将在GitHub和Hugging Face上开源我们的数据集。

英文摘要:

Enhancing the reasoning capabilities of Large Language Models (LLMs) with efficiency and scalability remains a fundamental challenge in artificial intelligence research. This paper presents a rigorous experimental investigation into how difficulty-aware staged reinforcement learning (RL) strategies can substantially improve LLM reasoning performance. Through systematic analysis, we demonstrate that strategically selecting training data according to well-defined difficulty levels markedly enhances RL optimization. Moreover, we introduce a staged training methodology, progressively exposing models to increasingly challenging tasks, further amplifying reasoning capabilities. Our findings reveal significant cross-domain benefits when simultaneously training models on mathematical reasoning and code generation tasks. Notably, our proposed approach enables a 1.5B parameter model to achieve an accuracy of 42.3\% on the AIME-2024 benchmark, 89.5\% on the MATH-500 benchmark. These results underscore the efficacy of our method in advancing the reasoning proficiency of LLMs. We will open-source our datasets on GitHub and Hugging Face.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑