arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2604.07941cs.CLcs.AIcs.LG
收藏

大语言模型后训练:一种统一的偏置学习与在线学习视角

Large Language Model Post-Training: A Unified View of Off-Policy and On-Policy Learning

Nankai University(南开大学);Huawei Technologies Ltd.(华为技术有限公司)

Shiwan Zhao, Zhihu Wang, Xuyang Zhao, Jiaming Zhou, Caiyue Xu, Chenfei Liu, Liting Zhang, Yuhang Jia, Yanzhe Zhang, Hualong Yu, Zichen Xu, Qicheng Li, Yong Qin

AI总结:

本文探讨了大语言模型后训练的统一框架,通过轨迹溯源划分偏置学习与在线学习两种模式,分析了支持扩展、策略重塑和行为固化等核心方法,强调系统设计协调性对进展的重要性。

摘要:

后训练已成为将预训练大语言模型(LLMs)转化为对齐、有能力且可部署系统的核心。近期进展涵盖监督微调(SFT)、偏好优化、强化学习(RL)、过程监督、验证器引导方法、蒸馏和多阶段流水线。然而,这些方法往往以标签或目标为组织方式,而非解决的行为瓶颈。本文调查认为,LLM后训练的最佳理解是结构化干预模型行为。我们首先按轨迹溯源组织该领域,定义两种主要模式:基于外部轨迹的偏置学习和基于学习者生成轨迹的在线学习。随后,通过两种反复出现的角色——有效支持扩展(使有用行为更可达)和策略重塑(改进已有可达区域内的行为)——以及互补的系统级角色——行为固化(在阶段和模型转换中保存、转移和摊薄有用行为)来解释方法。在此视角下,SFT可能服务于支持扩展或策略重塑;偏好优化通常为偏置重塑,但在线变种更接近学习者生成状态。在线RL常改善学习者生成状态上的行为,但更强的指导可使难以触及的推理路径可达。蒸馏通常被视为固化而非仅压缩,而混合流水线出现为协调的多阶段组合。总体而言,该框架有助于诊断后训练瓶颈并推理阶段组合,表明进展越来越依赖于协调的系统设计而非单一主导目标。

英文摘要:

Post-training has become central to turning pretrained large language models (LLMs) into aligned, capable, and deployable systems. Recent progress spans supervised fine-tuning (SFT), preference optimization, reinforcement learning (RL), process supervision, verifier-guided methods, distillation, and multi-stage pipelines. Yet these methods are often discussed in fragmented ways, organized by labels or objectives rather than by the behavioral bottlenecks they address. This survey argues that LLM post-training is best understood as structured intervention on model behavior. We organize the field first by trajectory provenance, which defines two primary regimes: off-policy learning on externally supplied trajectories and on-policy learning on learner-generated rollouts. We then interpret methods through two recurring roles -- effective support expansion, which makes useful behaviors more reachable, and policy reshaping, which improves behavior within already reachable regions -- together with a complementary systems-level role, behavioral consolidation, which preserves, transfers, and amortizes useful behavior across stages and model transitions. Under this view, SFT may serve either support expansion or policy reshaping; preference optimization is usually off-policy reshaping, though online variants move closer to learner-generated states. On-policy RL often improves behavior on learner-generated states, but stronger guidance can also make hard-to-reach reasoning paths reachable. Distillation is often better understood as consolidation rather than only compression, and hybrid pipelines emerge as coordinated multi-stage compositions. Overall, the framework helps diagnose post-training bottlenecks and reason about stage composition, suggesting that progress increasingly depends on coordinated systems design rather than any single dominant objective.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑