SFT-RL后训练中的困境:高SFT分数何时会产生误导以及可用的替代指标
Quagmires in SFT-RL Post-Training: When High SFT Scores Mislead and What to Use Instead
AI总结:
本研究揭示推理大模型SFT-RL后训练中高SFT分数无法可靠预测RL后性能的问题,提出泛化损失和Pass@large k作为更优代理指标,大幅提升预测精度,为后训练评估提供实用方案。
摘要:
在推理类大语言模型(LLM)的后训练流程中,当前主流实践分为两个独立阶段:监督微调(Supervised Fine-Tuning, SFT)和带可验证奖励的强化学习(Reinforcement Learning with Verifiable Rewards, RLVR,下文简称“RL”)。本研究对“高SFT分数能转化为RL后性能提升”这一认知提出质疑,并提供了大量反例证明该结论并不成立。研究发现,高SFT分数可能偏向更简单或同质化程度更高的数据,无法可靠预测后续的RL增益,也不能反映规模化后训练的实际效果。在部分案例中,在SFT性能提升的模型上进行RL训练,其最终结果甚至远差于在未经过SFT的基础模型上做RL训练的效果。\n本研究探索了替代评估指标,发现留出推理样本上的泛化损失(generalization loss)和Pass@large k性能是预测RL结果的强代理指标。研究通过GRPO算法训练了上百个最高达12B参数的SFT和RLVR模型,在7个数学基准测试上开展了最多256次重复的大规模评估,累计消耗超过100万GPU小时。实验涵盖Llama3、Mistral-Nemo、Qwen3等模型架构,以及多个当前最优的SFT/RL数据集。与直接基于RL前性能预测的方式相比,基于泛化损失和Pass@large k的预测精度大幅提升,R²系数和斯皮尔曼等级相关系数最高提升0.5(达到原来的2倍),这一发现对广泛应用场景具有很高的实用价值。例如,在大多数实验中,无论是仅做SFT还是SFT后接RL,用全部独特样本训练1个epoch的效果都不如用一半样本训练2个epoch;在相同SFT计算预算下,仅用短样本训练可能获得更好的SFT性能,但与用不同长度样本训练的模型相比,其RL后的结果通常更差。评估工具将开源发布。
英文摘要:
In post-training for reasoning Large Language Models (LLMs), the current state of practice trains LLMs in two independent stages: Supervised Fine-Tuning (SFT) and Reinforcement Learning with Verifiable Rewards (RLVR, shortened as ``RL'' below). In this work, we challenge whether high SFT scores translate to improved performance after RL. We provide extensive counter-examples where this is not true. We find high SFT scores can be biased toward simpler or more homogeneous data and are not reliably predictive of subsequent RL gains or scaled-up post-training effectiveness. In some cases, RL training on models with improved SFT performance could lead to substantially worse outcome compared to RL on the base model without SFT. We study alternative metrics and identify generalization loss on held-out reasoning examples and Pass@large k performance to provide strong proxies for the RL outcome. We trained hundreds of models up to 12B-parameter with SFT and RLVR via GRPO and ran extensive evaluations on 7 math benchmarks with up to 256 repetitions, spending $>$1M GPU hours. Experiments include models from Llama3, Mistral-Nemo, Qwen3 and multiple state-of-the-art SFT/RL datasets. Compared to directly predicting from pre-RL performance, prediction based on generalization loss and Pass@large k achieves substantial higher precision, improving $R^2$ coefficient and Spearman's rank correlation coefficient by up to 0.5 (2x). This provides strong utility for broad use cases. For example, in most experiments, we find SFT training on unique examples for a one epoch underperforms training on half examples for two epochs, either after SFT or SFT-then-RL; With the same SFT budget, training only on short examples may lead to better SFT performance, though, it often leads to worse outcome after RL compared to training on examples with varying lengths. Evaluation tool will be open-sourced.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。