arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2606.04466cs.CL
收藏

学习什么:小语言模型推理中SFT-then-RL的阶段特定数据集

Learning What to Learn: Stage-Specific Data Sets for SFT-then-RL in Small Language Model Reasoning

Tsinghua University(清华大学);National University of Singapore(新加坡国立大学);DiDi(滴滴出行);University of Electronic Science and Technology of China(电子科技大学)

Chongyang He, Rui Zhang, Zixuan Wang, Xin Li

AI总结:

提出一种难度感知的SFT-then-RL框架,通过阶段特定数据集(SFT阶段使用桥接机制,RL阶段使用批判微调)协调数据难度,提升小语言模型推理性能。

摘要:

后训练小语言模型(SLM)进行推理通常遵循SFT-then-RL流程,但现有工作很少考虑每个阶段应该学习什么数据。我们认为数据策略应与SFT和RL的不同角色对齐:SFT更适合获取尚未掌握的推理技能,而RL更适合巩固模型已部分掌握的技能。基于这一原则,我们提出了一种难度感知的SFT-then-RL框架,将训练数据组织成阶段特定的数据集。对于SFT阶段的困难样本,我们引入桥接机制,将教师生成的原始推理轨迹转化为SLM更易学习的监督信号。对于RL阶段仍未解决的困难样本,我们应用批判微调,将零奖励失败转化为诊断、修复和新的推理轨迹监督,用于下一SFT阶段。在两个SLM上跨越五个推理基准的实验表明,我们的方法在代表性SFT、蒸馏和RL基线上持续改进。我们的结果强调了协调SFT和RL之间数据难度对于有效SLM推理后训练的重要性。

英文摘要:

Post-training Small Language Models (SLMs) for reasoning typically follows an SFT-then-RL pipeline, yet existing work rarely considers what data should be learned at each stage. We argue that data strategy should be aligned with the distinct roles of SFT and RL: SFT is better suited for acquiring not-yet-mastered reasoning skills, while RL is better suited for consolidating skills that the model can already partially access. Based on this principle, we propose a difficulty-aware SFT-then-RL framework that organizes training data into stage-specific sets. For hard samples in the SFT stage, we introduce a Bridge mechanism that transforms raw teacher-generated reasoning traces into more learnable supervision for SLMs. For hard samples that remain unsolved during RL, we apply Critique Fine-Tuning by converting all-zero-reward failures into diagnostic, repair, and new reasoning trace supervision for the next SFT stage. Experiments on two SLMs across five reasoning benchmarks show that our method consistently improves over representative SFT, distillation, and RL baselines. Our results highlight the importance of coordinating data difficulty across SFT and RL for effective SLM reasoning post-training.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑