arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.01058cs.LGcs.AIcs.CL
收藏

好的SFT优化SFT,更好的SFT为强化学习做准备

Good SFT Optimizes for SFT, Better SFT Prepares for Reinforcement Learning

University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);New York University (Shanghai)(纽约大学(上海))

Dylan Zhang, Yufeng Xu, Haojin Wang, Qingzhi Chen, Hao Peng

AI总结:

针对当前SFT-RL流程中离线SFT数据分布与在线RL策略分布不匹配的问题,提出基于策略评估的离线学习损失重加权方法PEAR,通过重要性采样重加权SFT损失,提升后续RL训练效果。

摘要:

推理大语言模型的后训练是一个整体过程,通常包括离线SFT阶段和后续的在线强化学习(RL)阶段。然而,SFT通常被孤立地优化,仅追求最大化SFT性能。我们表明,在相同的RL训练后,从更强的SFT检查点初始化的模型可能显著劣于从较弱检查点初始化的模型。我们将此归因于当前SFT-RL流程中典型的错配:生成离线SFT数据的分布可能与在线RL期间优化的策略(该策略从其自身的rollout中学习)存在显著差异。我们提出PEAR(基于策略评估的离线学习损失重加权算法),这是一种在SFT阶段纠正此错配并让模型更好地为RL做准备的方法。PEAR使用重要性采样来重加权SFT损失,具有三种变体,分别在token、块和序列级别操作。它可以用于增强标准SFT目标,并且一旦收集到离线数据的概率,仅需很少的额外训练开销。我们在可验证推理游戏和数学推理任务上对Qwen 2.5和3以及DeepSeek蒸馏模型进行了控制实验。PEAR在标准SFT基础上持续提升了RL后性能,在AIME2025上pass@8增益高达14.6%。我们的结果表明,通过设计和评估SFT时考虑下游RL而非孤立进行,PEAR是迈向更全面的大语言模型后训练的有效一步。

英文摘要:

Post-training of reasoning LLMs is a holistic process that typically consists of an offline SFT stage followed by an online reinforcement learning (RL) stage. However, SFT is often optimized in isolation to maximize SFT performance alone. We show that, after identical RL training, models initialized from stronger SFT checkpoints can significantly underperform those initialized from weaker ones. We attribute this to a mismatch typical in current SFT-RL pipelines: the distribution that generates the offline SFT data can differ substantially from the policy optimized during online RL, which learns from its own rollouts. We propose PEAR (Policy Evaluation-inspired Algorithm for Offline Learning Loss Re-weighting), an SFT-stage method that corrects this mismatch and better prepares the model for RL. PEAR uses importance sampling to reweight the SFT loss, with three variants operating at the token, block, and sequence levels. It can be used to augment standard SFT objectives and incurs little additional training overhead once probabilities for the offline data are collected. We conduct controlled experiments on verifiable reasoning games and mathematical reasoning tasks on Qwen 2.5 and 3 and DeepSeek-distilled models. PEAR consistently improves post-RL performance over canonical SFT, with pass at 8 gains up to a 14.6 percent on AIME2025. Our results suggest that PEAR is an effective step toward more holistic LLM post-training by designing and evaluating SFT with downstream RL in mind rather than in isolation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑