Nanbeige4-3B 技术报告:探索小型语言模型的前沿
Nanbeige4-3B Technical Report: Exploring the Frontier of Small Language Models
AI总结:
Nanbeige4-3B通过FG-WSD调度器、DPD蒸馏和强化学习技术,实现了小型语言模型在性能和扩展定律上的突破。
摘要:
我们介绍了Nanbeige4-3B,一个小型但表现优异的语言模型家族。在23T高质量token上预训练,并在超过3000万种多样指令上微调,我们扩展了小型语言模型的扩展定律边界。在预训练阶段,我们设计了细粒度渐进稳定衰减(FG-WSD)训练调度器,通过逐步细化各阶段的数据混合来提升模型性能。在后训练阶段,为了提高SFT数据的质量,我们设计了一种联合机制,整合了反思生成优化和推理链重建,从而在复杂任务上取得了显著提升。随后,我们使用旗舰推理模型通过我们提出的双偏好蒸馏(DPD)方法对Nanbeige4-3B进行蒸馏,进一步提升了性能。最后,应用了多阶段强化学习阶段,利用可验证奖励和偏好建模来加强推理和人类对齐能力。广泛评估表明,Nanbeige4-3B不仅显著优于参数规模相当的模型,而且在广泛的基准测试中与更大规模的模型相抗衡。模型检查点可在https://huggingface.co/Nanbeige上获得。
英文摘要:
We present Nanbeige4-3B, a family of small-scale but high-performing language models. Pretrained on 23T high-quality tokens and finetuned on over 30 million diverse instructions, we extend the boundary of the scaling law for small language models. In pre-training, we design a Fine-Grained Warmup-Stable-Decay (FG-WSD) training scheduler, which progressively refines data mixtures across stages to boost model performance. In post-training, to improve the quality of the SFT data, we design a joint mechanism that integrates deliberative generation refinement and chain-of-thought reconstruction, yielding substantial gains on complex tasks. Following SFT, we employ our flagship reasoning model to distill Nanbeige4-3B through our proposed Dual Preference Distillation (DPD) method, which leads to further performance gains. Finally, a multi-stage reinforcement learning phase was applied, leveraging verifiable rewards and preference modeling to strengthen abilities on both reasoning and human alignment. Extensive evaluations show that Nanbeige4-3B not only significantly outperforms models of comparable parameter scale but also rivals much larger models across a wide range of benchmarks. The model checkpoints are available at https://huggingface.co/Nanbeige.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。