事后修复：对LLM事后训练数据质量和模型性能的比较研究
Fixing It in Post: A Comparative Study of LLM Post-Training Data Quality and Model Performance
AI总结：
本文通过对比两个开源事后训练数据集，提出了一种系统化编纂方法，生成性能更优的TuluTalk数据集，提升模型表现。
摘要：
近年来，大型语言模型（LLMs）的研究越来越多地聚焦于事后训练和与经过精心编纂以增强指令遵循、世界知识和专门技能的数据集对齐。然而，大多数用于领先开源和闭源LLM的事后训练数据集仍对公众不可用，其构建过程的信息有限。这种透明度的缺乏促使了最近开放源代码事后训练语料库的发展。虽然在这些开放替代品上训练可以产生与领先模型相当的性能，但系统性比较仍然具有挑战性，因为进行严格大规模比较的计算成本显著，因此基本上不存在。因此，在评估数据质量时，仍不清楚特定样本、任务类型或编纂策略如何影响下游性能。在本工作中，我们对两个著名的开放事后训练数据集：Tulu-3-SFT-Mix和SmolTalk进行了首次全面的并列分析。使用Magpie框架，我们对每个样本进行详细质量评估，包括对话结构（单轮 vs 多轮）、任务类别、输入质量和响应质量，并推导出揭示两个数据集之间结构和质量相似性和差异性的统计数据。基于这些见解，我们设计了一种系统化的编纂配方，产生了一个新的数据混合物TuluTalk，其样本数量比任一源数据集少14%，同时在关键基准上匹配或超过其性能。我们的发现为构建更有效的后续训练数据集提供了可行的见解，这些数据集在实际资源限制内提高了模型性能。为了支持未来研究，我们公开发布注释的源数据集和我们编纂的TuluTalk混合物。
英文摘要：
Recent work on large language models (LLMs) has increasingly focused on post-training and alignment with datasets curated to enhance instruction following, world knowledge, and specialized skills. However, most post-training datasets used in leading open- and closed-source LLMs remain inaccessible to the public, with limited information about their construction process. This lack of transparency has motivated the recent development of open-source post-training corpora. While training on these open alternatives can yield performance comparable to that of leading models, systematic comparisons remain challenging due to the significant computational cost of conducting them rigorously at scale, and are therefore largely absent. As a result, it remains unclear how specific samples, task types, or curation strategies influence downstream performance when assessing data quality. In this work, we conduct the first comprehensive side-by-side analysis of two prominent open post-training datasets: Tulu-3-SFT-Mix and SmolTalk. Using the Magpie framework, we annotate each sample with detailed quality metrics, including turn structure (single-turn vs. multi-turn), task category, input quality, and response quality, and we derive statistics that reveal structural and qualitative similarities and differences between the two datasets. Based on these insights, we design a principled curation recipe that produces a new data mixture, TuluTalk, which contains 14% fewer samples than either source dataset while matching or exceeding their performance on key benchmarks. Our findings offer actionable insights for constructing more effective post-training datasets that improve model performance within practical resource limits. To support future research, we publicly release both the annotated source datasets and our curated TuluTalk mixture.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
