SmolLM2：当小模型走向大规模——小语言模型的数据中心化训练
SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model
AI总结：
本文开发了 17 亿参数的小语言模型 SmolLM2，通过在 11 万亿 token 上进行多阶段数据中心化训练，并引入 FineMath、Stack-Edu 和 SmolTalk 等专门数据集，使其性能超越了 Qwen2.5-1.5B 和 Llama3.2-1B 等同类模型。
摘要：
尽管大语言模型在人工智能的众多应用中促成了突破性进展，但其固有的庞大规模使它们在计算上代价高昂，且难以部署在资源受限的环境中。在本文中，我们记录了 SmolLM2 的开发过程，这是一个最先进的“小型”（17 亿参数）语言模型（LM）。为了获得强大的性能，我们在约 11 万亿个 token 的数据上对 SmolLM2 进行了过度训练，采用多阶段训练过程，将网络文本与专门的数学、代码和指令遵循数据相混合。此外，在发现现有数据集规模过小或质量低下的阶段，我们引入了新的专门数据集（FineMath、Stack-Edu 和 SmolTalk）。为了为我们的设计决策提供依据，我们进行了小规模消融实验以及人工细化过程，该过程根据前一阶段的性能更新每个阶段的数据集混合比例。最终，我们证明 SmolLM2 优于其他近期的小型语言模型，包括 Qwen2.5-1.5B 和 Llama3.2-1B。为了促进未来关于语言模型开发以及小型语言模型应用的研究，我们发布了 SmolLM2 以及我们在本项目过程中准备的所有数据集。
英文摘要：
While large language models have facilitated breakthroughs in many applications of artificial intelligence, their inherent largeness makes them computationally expensive and challenging to deploy in resource-constrained settings. In this paper, we document the development of SmolLM2, a state-of-the-art "small" (1.7 billion parameter) language model (LM). To attain strong performance, we overtrain SmolLM2 on ~11 trillion tokens of data using a multi-stage training process that mixes web text with specialized math, code, and instruction-following data. We additionally introduce new specialized datasets (FineMath, Stack-Edu, and SmolTalk) at stages where we found existing datasets to be problematically small or low-quality. To inform our design decisions, we perform both small-scale ablations as well as a manual refinement process that updates the dataset mixing rates at each stage based on the performance at the previous stage. Ultimately, we demonstrate that SmolLM2 outperforms other recent small LMs including Qwen2.5-1.5B and Llama3.2-1B. To facilitate future research on LM development as well as applications of small LMs, we release both SmolLM2 as well as all of the datasets we prepared in the course of this project.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
