Post-training has become central to turning pretrained large language models (LLMs) into aligned, capable, and deployable systems. Recent progress spans supervised fine-tuning (SFT), preference optimization, reinforcement learning (RL), process supervision, verifier-guided methods, distillation, and multi-stage pipelines. Yet these methods are often discussed in fragmented ways, organized by labels or objectives rather than by the behavioral bottlenecks they address. This survey argues that LLM post-training is best understood as structured intervention on model behavior. We organize the field first by trajectory provenance, which defines two primary regimes: off-policy learning on externally supplied trajectories and on-policy learning on learner-generated rollouts. We then interpret methods through two recurring roles -- effective support expansion, which makes useful behaviors more reachable, and policy reshaping, which improves behavior within already reachable regions -- together with a complementary systems-level role, behavioral consolidation, which preserves, transfers, and amortizes useful behavior across stages and model transitions. Under this view, SFT may serve either support expansion or policy reshaping; preference optimization is usually off-policy reshaping, though online variants move closer to learner-generated states. On-policy RL often improves behavior on learner-generated states, but stronger guidance can also make hard-to-reach reasoning paths reachable. Distillation is often better understood as consolidation rather than only compression, and hybrid pipelines emerge as coordinated multi-stage compositions. Overall, the framework helps diagnose post-training bottlenecks and reason about stage composition, suggesting that progress increasingly depends on coordinated systems design rather than any single dominant objective.
GVPO: Group Variance Policy Optimization for Large Language Model Post-Training
Thrust of Artificial Intelligence, Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州)人工智能研究所);Zuoyebang Education Technology(佐业邦教育科技);Department of Computer Science and Engineering, HKUST(香港科技大学计算机科学与工程系)
Actor-Curator: Co-adaptive Curriculum Learning via Policy-Improvement Bandits for RL Post-Training
University of Illinois Chicago(伊利诺伊大学香槟分校);Caltech(加州理工学院);RPI(罗切斯特理工学院);MBZUAI(澳门大学人工智能研究院);University of Chicago(芝加哥大学);NEC Laboratories America(NEC美国实验室)
On-Policy Parameter Update Direction Underlies Generalization in LLM Post-Training
State Key Lab. of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所人工智能安全国家重点实验室);University of Chinese Academy of Sciences(中国科学院大学);Meituan(美团)