arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2604.01702cs.CL
收藏

在长链式思维监督微调中推理模式在泛化差异中的作用

On the Role of Reasoning Patterns in the Generalization Discrepancy of Long Chain-of-Thought Supervised Fine-Tuning

University of Science and Technology of China(中国科学技术大学);Meituan LongCat Team(美团龙猫团队);City University of Hong Kong(香港城市大学);Zhejiang University(浙江大学)

Zhaoyi Li, Xiangyu Xi, Zhengyu Chen, Wei Wang, Gangwei Jiang, Ranran Shen, Linqi Song, Ying Wei, Defu Lian

AI总结:

本文研究不同来源的长链式思维轨迹对模型泛化性能的影响,发现训练损失低并不一定带来更好的泛化能力,提出过滤频繁分支轨迹以提升SFT泛化性能。

摘要:

监督微调(SFT)在长链式思维(CoT)轨迹上已成为构建大推理模型的关键阶段。然而,不同来源的CoT轨迹如何影响模型的泛化性能仍是一个开放问题。本文通过比较两个来源的验证CoT轨迹(由DeepSeek-R1-0528和gpt-oss-120b生成)进行研究,其问题集被控制为相同。尽管两者表现相似,我们发现训练损失低并不一定带来更好的泛化能力。在DeepSeek-R1-0528数据上进行SFT获得显著较低的训练损失,但其在推理基准上的泛化性能却比gpt-oss-120b差。通过多方面的分析,我们发现推理模式存在差异:gpt-oss-120b表现出高度收敛和演绎轨迹,而DeepSeek-R1-0528则倾向于发散和分支繁多的探索模式。因此,使用DeepSeek-R1数据训练的模型继承了低效的探索行为,常陷入冗余的探索分支,阻碍正确解的达成。基于此,我们提出过滤频繁分支轨迹以提高SFT泛化性能。实验显示,训练在选定的DeepSeek-R1-0528子集上,意外在AIME25上提升了5.1%的推理性能,在BeyondAIME上提升了5.5%,在五个基准上平均提升了3.6%。

英文摘要:

Supervised Fine-Tuning (SFT) on long Chain-of-Thought (CoT) trajectories has become a pivotal phase in building large reasoning models. However, how CoT trajectories from different sources influence the generalization performance of models remains an open question. In this paper, we conduct a comparative study using two sources of verified CoT trajectories generated by two competing models, \texttt{DeepSeek-R1-0528} and \texttt{gpt-oss-120b}, with their problem sets controlled to be identical. Despite their comparable performance, we uncover a striking paradox: lower training loss does not translate to better generalization. SFT on \texttt{DeepSeek-R1-0528} data achieves remarkably lower training loss, yet exhibits significantly worse generalization performance on reasoning benchmarks compared to those trained on \texttt{gpt-oss-120b}. To understand this paradox, we perform a multi-faceted analysis probing token-level SFT loss and step-level reasoning behaviors. Our analysis reveals a difference in reasoning patterns. \texttt{gpt-oss-120b} exhibits highly convergent and deductive trajectories, whereas \texttt{DeepSeek-R1-0528} favors a divergent and branch-heavy exploration pattern. Consequently, models trained with \texttt{DeepSeek-R1} data inherit inefficient exploration behaviors, often getting trapped in redundant exploratory branches that hinder them from reaching correct solutions. Building upon this insight, we propose a simple yet effective remedy of filtering out frequently branching trajectories to improve the generalization of SFT. Experiments show that training on selected \texttt{DeepSeek-R1-0528} subsets surprisingly improves reasoning performance by up to 5.1% on AIME25, 5.5% on BeyondAIME, and on average 3.6% on five benchmarks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑