arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2607.01232cs.LGcs.CL
收藏

一层就够了吗?训练单个Transformer层可以匹配全参数RL训练

Is One Layer Enough? Training A Single Transformer Layer Can Match Full-Parameter RL Training

University of Minnesota(明尼苏达大学);Peking University(北京大学);Amazon(亚马逊)

Zijian Zhang, Rizhen Hu, Athanasios Glentis, Dawei Li, Chung-Yiu Yau, Hongzhou Lin, Mingyi Hong

AI总结:

发现RL训练收益高度集中在少数Transformer层,仅训练单层即可恢复大部分全参数RL收益,且高贡献层集中在模型中部。

摘要:

强化学习(RL)已成为大型语言模型(LLM)后训练的核心组成部分,但关于RL适应如何分布在Transformer各层中的了解甚少。现有方法通常统一更新所有模型参数,隐含地假设每一层对RL后训练获得的收益贡献相似。在这项工作中,我们通过系统的逐层RL训练研究挑战了这一假设。令人惊讶的是,我们发现训练单个Transformer层可以恢复全参数RL训练所获得的大部分收益,在某些情况下甚至超越它。为了量化这一现象,我们引入了层贡献量,它衡量了单独训练一层所恢复的全RL改进的比例。在跨越两个模型家族(Qwen3、Qwen2.5)、三种RL算法(GRPO、GiGPO、Dr. GRPO)以及包括数学推理、代码生成和智能体决策在内的多个任务领域的七个模型中,我们观察到一个非常稳定的模式:RL收益高度集中在一小部分,甚至在许多情况下是单个Transformer层中。更引人注目的是,相同的结构模式一致出现:高贡献层集中在Transformer堆栈的中部,而靠近输入和输出端的层贡献显著较少。由此产生的层排名在数据集、任务、模型家族和RL算法之间保持强相关性。

英文摘要:

Reinforcement learning (RL) has become a central component of post-training large language models (LLMs), yet little is understood about how RL adaptation is distributed across transformer layers. Existing approaches typically update all model parameters uniformly, implicitly assuming that every layer contributes similarly to the gains obtained during RL post-training. In this work, we challenge this assumption through a systematic layer-wise study of RL training. Surprisingly, we find that training a single transformer layer can recover most of the gains achieved by full-parameter RL training, and in some cases even surpass it. To quantify this phenomenon, we introduce the quantity layer contribution, which measures the fraction of full RL improvement recovered by training a layer in isolation. Across seven models spanning two model families (Qwen3, Qwen2.5), three RL algorithms (GRPO, GiGPO, Dr. GRPO), and multiple task domains including mathematical reasoning, code generation, and agentic decision-making, we observe a remarkably stable pattern: RL gains are highly concentrated in a small subset of, and in many cases even a single, transformer layers. More strikingly, the same structural pattern consistently emerges: high-contribution layers concentrate in the middle of the transformer stack, while layers near the input and output ends contribute substantially less. The resulting layer rankings remain strongly correlated across datasets, tasks, model families, and RL algorithms.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑