如何微调一个推理模型？一种教师-学生合作框架来合成学生一致的SFT数据
How to Fine-Tune a Reasoning Model? A Teacher-Student Cooperation Framework to Synthesize Student-Consistent SFT Data
AI总结：
本文提出TESSY框架，通过交替生成风格和非风格标记，解决教师生成数据与学生分布之间的风格差异问题，提升推理模型性能。
摘要：
一种广泛采用的模型增强策略是使用更强模型生成的合成数据进行监督微调（SFT）。然而，对于新兴的推理模型如Qwen3-8B，这种方法往往无法提升推理能力，甚至可能导致性能大幅下降。在本工作中，我们识别出教师生成数据与学生分布之间的显著风格差异是影响SFT的主要因素。为弥合这一差距，我们提出了教师-学生合作数据合成框架（TESSY），通过交替生成教师和学生模型的风格和非风格标记，从而生成继承教师先进推理能力且保持学生分布风格一致的合成序列。在使用GPT-OSS-120B作为教师进行代码生成实验中，对Qwen3-8B进行教师生成数据微调导致在LiveCodeBench-Pro上性能下降3.25%，在OJBench上下降10.02%，而TESSY在LiveCodeBench-Pro上提升11.25%，在OJBench上提升6.68%。
英文摘要：
A widely adopted strategy for model enhancement is to use synthetic data generated by a stronger model for supervised fine-tuning (SFT). However, for emerging reasoning models like Qwen3-8B, this approach often fails to improve reasoning capabilities and can even lead to a substantial drop in performance. In this work, we identify substantial stylistic divergence between teacher generated data and the distribution of student as a major factor impacting SFT. To bridge this gap, we propose a Teacher-Student Cooperation Data Synthesis framework (TESSY), which interleaves teacher and student models to alternately generate style and non-style tokens. Consequently, TESSY produces synthetic sequences that inherit the advanced reasoning capabilities of the teacher while maintaining stylistic consistency with the distribution of the student. In experiments on code generation using GPT-OSS-120B as the teacher, fine-tuning Qwen3-8B on teacher-generated data leads to performance drops of 3.25% on LiveCodeBench-Pro and 10.02% on OJBench, whereas TESSY achieves improvements of 11.25% and 6.68%.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
