arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2601.09088cs.LGcs.CL
收藏

面向优越长链推理的分布对齐序列蒸馏

Distribution-Aligned Sequence Distillation for Superior Long-CoT Reasoning

Alibaba Cloud Computing(阿里巴巴云 computing)

Shaotian Yan, Kaiyuan Liu, Chen Shen, Bing Wang, Sinan Fan, Jun Zhang, Yue Wu, Zheng Wang, Jieping Ye

AI总结:

本文提出DASD-4B-Thinking模型,通过改进序列蒸馏方法,实现高效推理性能,仅用448K样本达到SOTA效果。

摘要:

在本报告中,我们介绍了DASD-4B-Thinking,一个轻量但能力强大且完全开源的推理模型。它在数学、科学推理和代码生成等具有挑战性的基准测试中,实现了与同类开源模型相当的SOTA性能,甚至在某些方面超过了几个更大的模型。我们首先批判性地重新审视社区广泛采用的蒸馏范式:基于教师生成响应的有监督微调(SFT),也称为序列级蒸馏。尽管一系列后续工作展示了该方案的高效性和强大的实证性能,但它们主要基于SFT视角。因此,这些方法主要集中在设计启发式规则来过滤SFT数据,而很大程度上忽略了蒸馏的核心原理——使学生模型学习教师的完整输出分布,从而继承其泛化能力。具体而言,我们识别出当前实践中存在三个关键局限:i)教师序列级分布表示不足;ii)教师输出分布与学生学习能力之间的不匹配;iii)由于教师强制训练与自回归推理之间的暴露偏差。总而言之,这些不足反映了蒸馏过程中教师-学生互动的系统性缺失,导致蒸馏的本质被低估。为了解决这些问题,我们提出了一系列方法论创新,这些创新共同构成了一个增强的序列级蒸馏训练流水线。值得注意的是,DASD-4B-Thinking仅使用448K训练样本就获得了具有竞争力的结果——比大多数现有开源项目使用的样本数量少一个数量级。为了支持社区研究,我们公开发布了我们的模型和训练数据集。

英文摘要:

In this report, we introduce DASD-4B-Thinking, a lightweight yet highly capable, fully open-source reasoning model. It achieves SOTA performance among open-source models of comparable scale across challenging benchmarks in mathematics, scientific reasoning, and code generation -- even outperforming several larger models. We begin by critically reexamining a widely adopted distillation paradigm in the community: SFT on teacher-generated responses, also known as sequence-level distillation. Although a series of recent works following this scheme have demonstrated remarkable efficiency and strong empirical performance, they are primarily grounded in the SFT perspective. Consequently, these approaches focus predominantly on designing heuristic rules for SFT data filtering, while largely overlooking the core principle of distillation itself -- enabling the student model to learn the teacher's full output distribution so as to inherit its generalization capability. Specifically, we identify three critical limitations in current practice: i) Inadequate representation of the teacher's sequence-level distribution; ii) Misalignment between the teacher's output distribution and the student's learning capacity; and iii) Exposure bias arising from teacher-forced training versus autoregressive inference. In summary, these shortcomings reflect a systemic absence of explicit teacher-student interaction throughout the distillation process, leaving the essence of distillation underexploited. To address these issues, we propose several methodological innovations that collectively form an enhanced sequence-level distillation training pipeline. Remarkably, DASD-4B-Thinking obtains competitive results using only 448K training samples -- an order of magnitude fewer than those employed by most existing open-source efforts. To support community research, we publicly release our models and the training dataset.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑