arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.03075cs.CL
收藏

ReMiT: 通过强化学习指导的迭代大语言模型进化

ReMiT: RL-Guided Mid-Training for Iterative LLM Evolution

Shanghai Jiao Tong University(上海交通大学);Tencent Youtu Lab(腾讯优图实验室)

Junjie Huang, Jiarui Qin, Di Yin, Weiwen Liu, Yong Yu, Xing Sun, Weinan Zhang

AI总结:

ReMiT通过强化学习指导中训练过程,实现大语言模型的迭代进化,提升模型在数学、代码和推理任务上的表现。

摘要:

标准的大语言模型(LLM)训练流程通常是单向的,从预训练到后训练逐步推进。然而,双向过程的潜力——即后训练的洞察可以反向提升预训练基础——仍未被探索。我们旨在建立一个自我强化的飞轮:一个循环,其中强化学习(RL)调优的模型增强基础模型,而基础模型的提升又会增强后续后训练的表现,无需特别训练的教师或参考模型。为此,我们分析了训练动态,并确定了中训练(退火)阶段作为模型能力关键转折点。该阶段通常发生在预训练结束时,利用高质量语料库并在学习率迅速衰减的情况下进行。基于这一见解,我们引入了ReMiT(强化学习指导的中训练)。具体而言,ReMiT利用RL调优模型的推理先验,动态重新加权中训练阶段的token,优先考虑那些对推理关键的token。实证结果表明,ReMiT在10个预训练基准上平均提升了3%,涵盖数学、代码和一般推理,且在后训练流程中持续保持超过2%的提升。这些结果验证了迭代反馈循环,使LLM能够实现连续且自我强化的进化。

英文摘要:

Standard training pipelines for large language models (LLMs) are typically unidirectional, progressing from pre-training to post-training. However, the potential for a bidirectional process--where insights from post-training retroactively improve the pre-trained foundation--remains unexplored. We aim to establish a self-reinforcing flywheel: a cycle in which reinforcement learning (RL)-tuned model strengthens the base model, which in turn enhances subsequent post-training performance, requiring no specially trained teacher or reference model. To realize this, we analyze training dynamics and identify the mid-training (annealing) phase as a critical turning point for model capabilities. This phase typically occurs at the end of pre-training, utilizing high-quality corpora under a rapidly decaying learning rate. Building upon this insight, we introduce ReMiT (Reinforcement Learning-Guided Mid-Training). Specifically, ReMiT leverages the reasoning priors of RL-tuned models to dynamically reweight tokens during the mid-training phase, prioritizing those pivotal for reasoning. Empirically, ReMiT achieves an average improvement of 3\% on 10 pre-training benchmarks, spanning math, code, and general reasoning, and sustains these gains by over 2\% throughout the post-training pipeline. These results validate an iterative feedback loop, enabling continuous and self-reinforcing evolution of LLMs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑