Recent mixed-policy optimization methods for LLM reasoning that interleave or blend supervised and reinforcement learning signals report improvements over the standard SFT-then-RL pipeline. We show that numerous recently published research papers rely on a faulty baseline caused by two distinct bugs: a CPU-offloaded optimizer bug in DeepSpeed that silently drops intermediate micro-batches during gradient accumulation (affecting multiple downstream frameworks including TRL, OpenRLHF and Llama-Factory), and a loss aggregation bug in OpenRLHF that incorrectly weights per-mini-batch losses. Together they suppress SFT performance, with the optimizer bug accounting for most of the gap and the loss aggregation bug contributing a smaller additional effect. Once corrected, the standard SFT-then-RL pipeline surpasses every published mixed-policy method we evaluate by +3.8 points on math benchmarks with Qwen2.5-Math-7B and by +22.2 points with Llama-3.1-8B. Even a truncated variant with just 50 RL steps outperforms mixed-policy methods on math benchmarks while using fewer FLOPs.
Shanghai Jiao Tong University(上海交通大学);Shanghai Innovation Institute(上海创新研究院);Large Language Model Department, Tencent(腾讯大语言模型部);University of Macau(澳门大学)
Decoupling KL and Trajectories: A Unified Perspective for SFT, DAgger, Offline RL, and OPD in LLM Distillation
Eastern Institute of Technology(东技术院);The Hong Kong Polytechnic University(香港理工大学);Shanghai Jiao Tong University(上海交通大学);Thrust of Artificial Intelligence, The Hong Kong University of Science and Technology (Guangzhou)(人工智能 thrust,香港科学与技术大学(广州))
Scaling Near-Optimal SFT-RL Annotation Budget Allocation from Small to Large LLMs
National University of Singapore(新加坡国立大学);Agency for Science, Technology, and Research (A*STAR)(新加坡科技研究局);Singapore-MIT Alliance for Research and Technology Centre(新加坡-麻省理工研究与技术联盟中心)
Bridging SFT and RL: Dynamic Policy Optimization for Robust Reasoning
Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);Intelligent System Department, Zhongxing Telecom Equipment (ZTE)(中兴通讯股份有限公司智能系统部);Institute of Advanced Interdisciplinary Studies, Chongqing University of Posts and Telecommunications(重庆邮电大学前沿交叉研究院)
RLHF in an SFT Way: From Optimal Solution to Reward-Weighted Alignment
Shenzhen Research Institute of Big Data(深圳大数据研究院);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Qwen LLM Application Team, Alibaba(阿里巴巴Qwen大模型应用团队);Stanford University(斯坦福大学)
RASFT: Rollout-Adaptive Supervised Fine-Tuning for Reasoning
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);New Jersey Institute of Technology(新泽西理工学院);Institute of Computing Technology, CAS(中国科学院计算技术研究所)
When RL Fails after SFT: Rejuvenating Model Plasticity for Robust SFT-to-RL Handoff
Hong Kong University of Science and Technology(香港科技大学);Zhejiang University(浙江大学);State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA(多模态人工智能系统国家重点实验室,CASIA)
LP-SFT: Keeping Plausible Alternatives Alive in Supervised Fine-Tuning
School of Mathematical Sciences, Peking University(北京大学数学科学学院);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);College of Computing, Georgia Institute of Technology(佐治亚理工学院计算学院)
PolicyShiftGuard: Benchmarking and Improving Policy-Adaptive Image Guardrails
Fudan University(复旦大学);Tongji University(同济大学);Virtue AI(美德人工智能公司);The Chinese University of Hong Kong(香港中文大学);University of Chicago(芝加哥大学);University of Illinois, Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)
Draft-OPD: On-Policy Distillation for Speculative Draft Models
Shanghai Jiao Tong University(上海交通大学);Shanghai AI Laboratory(上海人工智能实验室);Tsinghua University(清华大学);The Chinese University of Hong Kong(香港中文大学);Peking University(北京大学);Zhejiang University(浙江大学)