arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2604.23747cs.LGcs.AIcs.CL
收藏

在LLM推理中,SFT-然后-RL优于混合策略方法

SFT-then-RL Outperforms Mixed-Policy Methods for LLM Reasoning

ETH AI Center, ETH Zürich(苏黎世联邦理工学院人工智能中心);EPFL(瑞士联邦理工学院);Allen Institute for AI(人工智能研究所)

Alexis Limozin, Eduard Durech, Torsten Hoefler, Imanol Schlag, Valentina Pyatkin

AI总结:

本文指出,混合策略方法的性能提升源于两个错误:DeepSpeed优化器在梯度累积中丢失中间微批次,以及OpenRLHF损失聚合错误加权。修正后,标准SFT-然后-RL流程在数学基准上超越了所有混合策略方法。

摘要:

最近的混合策略优化方法在LLM推理中交替或混合监督学习和强化学习信号,报告了比标准SFT-然后-RL流程更好的改进。我们发现,许多最近发表的论文依赖于一个错误的基准,这是由两个不同的错误引起的:DeepSpeed中的CPU卸载优化器错误在梯度累积中静默地丢失中间微批次(影响多个下游框架,包括TRL、OpenRLHF和Llama-Factory),以及OpenRLHF中的损失聚合错误错误加权每个小批次的损失。它们抑制了SFT性能,其中优化器错误占大部分差距,而损失聚合错误贡献了较小的额外影响。一旦修正,标准SFT-然后-RL流程在Qwen2.5-Math-7B上比我们评估的所有混合策略方法高出+3.8点,在Llama-3.1-8B上高出+22.2点。即使是一个截断的变体,仅使用50次RL步骤,在数学基准上也优于混合策略方法,同时使用更少的FLOPs。

英文摘要:

Recent mixed-policy optimization methods for LLM reasoning that interleave or blend supervised and reinforcement learning signals report improvements over the standard SFT-then-RL pipeline. We show that numerous recently published research papers rely on a faulty baseline caused by two distinct bugs: a CPU-offloaded optimizer bug in DeepSpeed that silently drops intermediate micro-batches during gradient accumulation (affecting multiple downstream frameworks including TRL, OpenRLHF and Llama-Factory), and a loss aggregation bug in OpenRLHF that incorrectly weights per-mini-batch losses. Together they suppress SFT performance, with the optimizer bug accounting for most of the gap and the loss aggregation bug contributing a smaller additional effect. Once corrected, the standard SFT-then-RL pipeline surpasses every published mixed-policy method we evaluate by +3.8 points on math benchmarks with Qwen2.5-Math-7B and by +22.2 points with Llama-3.1-8B. Even a truncated variant with just 50 RL steps outperforms mixed-policy methods on math benchmarks while using fewer FLOPs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑