Post-training of reasoning LLMs is a holistic process that typically consists of an offline SFT stage followed by an online reinforcement learning (RL) stage. However, SFT is often optimized in isolation to maximize SFT performance alone.
We show that, after identical RL training, models initialized from stronger SFT checkpoints can significantly underperform those initialized from weaker ones. We attribute this to a mismatch typical in current SFT-RL pipelines: the distribution that generates the offline SFT data can differ substantially from the policy optimized during online RL, which learns from its own rollouts.
We propose PEAR (Policy Evaluation-inspired Algorithm for Offline Learning Loss Re-weighting), an SFT-stage method that corrects this mismatch and better prepares the model for RL. PEAR uses importance sampling to reweight the SFT loss, with three variants operating at the token, block, and sequence levels. It can be used to augment standard SFT objectives and incurs little additional training overhead once probabilities for the offline data are collected.
We conduct controlled experiments on verifiable reasoning games and mathematical reasoning tasks on Qwen 2.5 and 3 and DeepSeek-distilled models. PEAR consistently improves post-RL performance over canonical SFT, with pass at 8 gains up to a 14.6 percent on AIME2025. Our results suggest that PEAR is an effective step toward more holistic LLM post-training by designing and evaluating SFT with downstream RL in mind rather than in isolation.
When RL Fails after SFT: Rejuvenating Model Plasticity for Robust SFT-to-RL Handoff
Hong Kong University of Science and Technology(香港科技大学);Zhejiang University(浙江大学);State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA(多模态人工智能系统国家重点实验室,CASIA)
Benchmark Success, Clinical Failure: When Reinforcement Learning Optimizes for Benchmarks, Not Patients
Fraunhofer IAIS(弗劳恩霍夫人工智能研究所);University of Bonn(波恩大学);Lamarr Institute(拉马尔研究所);Department of Health Queensland(昆士兰健康部);Griffith University(格里菲斯大学);University Hospital Bonn(波恩大学医院)
A Practical Two-Stage Recipe for Mathematical LLMs: Maximizing Accuracy with SFT and Efficiency with Reinforcement Learning
Aillis Inc.(Aillis公司);Department of Health Policy(健康政策部门);Public Health, Graduate School of Pharmaceutical Sciences, The University of Tokyo, Tokyo, Japan(公共卫生,药学研究生院,东京大学,东京,日本);Rist Inc.(Rist公司)
SRFT: A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoning
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Meituan(美团);Shanghai Jiao Tong University(上海交通大学)
Architecture-Aware Reinforcement Learning Makes Sliding-Window Attention Competitive in Math Reasoning
Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University(上海智能自主系统研究院,同济大学);Shanghai AI Laboratory(上海人工智能实验室);Hong Kong University of Science and Technology (Guangzhou)(香港理工大学(广州))
Offline Exploration-Aware Fine-Tuning for Long-Chain Mathematical Reasoning
NLP Lab, School of Computer Science and Engineering, Northeastern University, Shenyang, China(东北大学计算机科学与工程学院自然语言处理实验室,中国沈阳);Pattern Recognition Center, WeChat AI, Tencent Inc, China(腾讯公司微信人工智能部门模式识别中心,中国)
Distill Not Only Data but Also Rewards: Can Smaller Language Models Surpass Larger Ones?
Eindhoven University of Technology(埃因霍温理工大学);Microsoft(微软公司);University of Liverpool(利物浦大学);King’s College London(伦敦国王学院);Fudan University(复旦大学);University College London(伦敦大学学院)
Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questions
Peking University(北京大学);Zhongguancun Academy(中关村学院);Beijing Key Laboratory of Data Intelligence and Security (Peking University)(北京数据智能与安全重点实验室(北京大学));Beijing Key Laboratory of Software and Hardware Cooperative Artificial Intelligence Systems(北京软件与硬件协同人工智能系统重点实验室)
CSRP: Chain-of-Thought Reasoning for Chinese Text Correction via Reinforcement Learning with Efficiency-Aware Rewards
School of Computer Science and Technology, East China Normal University(东华大学计算机科学与技术学院);Shanghai Institute of Artificial Intelligence for Education, East China Normal University(东华大学教育人工智能研究所)
On-Policy Supervised Fine-Tuning for Efficient Reasoning
Department of Computing, The Hong Kong Polytechnic University(香港理工大学计算机系);Eastern Institute of Technology, Ningbo(宁波工程技术学院);Shanghai Jiao Tong University(上海交通大学);Ludwig Maximilian University of Munich(慕尼黑大学);Tencent Hunyuan / AI Lab(腾讯混元/AI实验室)
Tourism Question Answer System in Indian Language using Domain-Adapted Foundation Models
Department of Linguistics Banaras Hindu University(语言学系班加罗尔 Hindu 大学);Department of Humanistic Studies Indian Institute of Technology (BHU)(人文科学系印度理工学院(BHU));Department of Computer Science and Engineering Indian Institute of Technology Bhilai(计算机科学与工程系印度理工学院 Bhilai)
REAL: Regression-Aware Reinforcement Learning for LLM-as-a-Judge
University of California, Los Angeles(加州大学洛杉矶分校);The University of Texas at Austin(得克萨斯大学奥斯汀分校);Google Research Now at Google DeepMind(谷歌研究 现在在谷歌深Mind)