Tulu 3：推动开放语言模型后训练的前沿
Tulu 3: Pushing Frontiers in Open Language Model Post-Training
AI总结：
针对开放语言模型后训练方案落后于专有方案的问题，提出完全开放的Tulu 3模型系列，公开其数据、代码和训练方案，采用SFT、DPO及新方法RLVR，性能超多款主流模型，还提供多任务评估方案和完整复现指南。
摘要：
语言模型后训练被应用于改进行为并解锁近期各类语言模型的新技能，但应用这些技术的开放方案落后于专有方案。后训练的基础训练数据和方案既是该领域最重要的部分，也是透明度最低的部分。为弥合这一差距，我们引入Tulu 3——一系列完全开放的最先进后训练模型，同时公开其数据、代码和训练方案，作为现代后训练技术的综合指南。Tulu 3基于Llama 3.1基础模型构建，性能超过Llama 3.1、Qwen 2.5、Mistral的指令版本，甚至超过GPT-4o-mini和Claude 3.5-Haiku等闭源模型。我们模型的训练算法包括监督微调（SFT）、直接偏好优化（DPO）以及一种名为可验证奖励强化学习（RLVR）的新方法。通过Tulu 3，我们引入了后训练方案的多任务评估方案（含开发和未见评估）、标准基准实现，以及对现有开放数据集在上述基准上的大量去污染处理。我们最后分析并讨论了未可靠提升性能的训练方法。除Tulu 3模型权重和演示外，我们还发布了完整方案——包括针对多种核心技能的数据集、用于数据整理和评估的稳健工具包、训练代码和基础设施，以及最重要的：一份用于复现并进一步将Tulu 3方法适配到更多领域的详细报告。
英文摘要：
Language model post-training is applied to refine behaviors and unlock new skills across a wide range of recent language models, but open recipes for applying these techniques lag behind proprietary ones. The underlying training data and recipes for post-training are simultaneously the most important pieces of the puzzle and the portion with the least transparency. To bridge this gap, we introduce Tulu 3, a family of fully-open state-of-the-art post-trained models, alongside its data, code, and training recipes, serving as a comprehensive guide for modern post-training techniques. Tulu 3, which builds on Llama 3.1 base models, achieves results surpassing the instruct versions of Llama 3.1, Qwen 2.5, Mistral, and even closed models such as GPT-4o-mini and Claude 3.5-Haiku. The training algorithms for our models include supervised finetuning (SFT), Direct Preference Optimization (DPO), and a novel method we call Reinforcement Learning with Verifiable Rewards (RLVR). With Tulu 3, we introduce a multi-task evaluation scheme for post-training recipes with development and unseen evaluations, standard benchmark implementations, and substantial decontamination of existing open datasets on said benchmarks. We conclude with analysis and discussion of training methods that did not reliably improve performance. In addition to the Tulu 3 model weights and demo, we release the complete recipe -- including datasets for diverse core skills, a robust toolkit for data curation and evaluation, the training code and infrastructure, and, most importantly, a detailed report for reproducing and further adapting the Tulu 3 approach to more domains.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
