Phi-4 技术报告
Phi-4 Technical Report
AI总结：
本研究推出140亿参数的phi-4语言模型，通过在训练全程策略性融入合成数据、优化训练课程与后训练方案，在架构改动极小的情况下，于STEM问答和推理基准上表现优异，甚至超越GPT-4教师模型。
摘要：
我们推出phi-4，这是一款拥有140亿参数的语言模型，其训练方案核心聚焦于数据质量。与大多数预训练主要基于网页内容或代码等有机数据源的语言模型不同，phi-4在整个训练过程中策略性地融入了合成数据。Phi系列的前代模型主要是蒸馏教师模型（具体为GPT-4）的能力，而phi-4在侧重STEM的问答能力上大幅超越了其教师模型，这证明我们的数据生成和后训练技术的效果超越了蒸馏。尽管phi-4的架构相比phi-3仅有极小改动，但得益于改进的数据、训练课程以及后训练方案的创新，phi-4在同规模模型中实现了强劲性能，尤其是在侧重推理的基准测试上。
英文摘要：
We present phi-4, a 14-billion parameter language model developed with a training recipe that is centrally focused on data quality. Unlike most language models, where pre-training is based primarily on organic data sources such as web content or code, phi-4 strategically incorporates synthetic data throughout the training process. While previous models in the Phi family largely distill the capabilities of a teacher model (specifically GPT-4), phi-4 substantially surpasses its teacher model on STEM-focused QA capabilities, giving evidence that our data-generation and post-training techniques go beyond distillation. Despite minimal changes to the phi-3 architecture, phi-4 achieves strong performance relative to its size -- especially on reasoning-focused benchmarks -- due to improved data, training curriculum, and innovations in the post-training scheme.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
