arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.01293cs.LGcs.AIstat.ML
收藏

数据质量和协同效应在预训练和后训练推理模型中的理论视角

Theoretical Perspectives on Data Quality and Synergistic Effects in Pre- and Post-Training Reasoning Models

University of Southern California(南加州大学);University of California Los Angeles(加州大学洛杉矶分校);Google Research(谷歌研究院)

Adel Javanmard, Baharan Mirzasoleiman, Vahab Mirrokni

AI总结:

本文通过理论分析揭示了预训练和后训练模型中数据质量与协同效应的关键机制,发现平衡预训练数据能激活潜在能力,SFT在挑战性小样本上表现最佳,而RL在大规模非困难数据上更有效。

摘要:

大型语言模型(LLMs)在大规模数据集上进行预训练,然后通过监督微调(SFT)或强化学习(RL)进行指令微调。最佳实践强调使用大量、多样化的预训练数据,而后训练则有所不同:SFT依赖于较小的高质量数据集,而RL则受益于规模,更多的反馈量往往超过标签质量。然而,仍不清楚为何预训练和RL需要大量数据,为何SFT在较小的数据集上表现优异,以及什么定义了高质量的SFT数据。在本工作中,我们理论分析了在上下文权重预测任务上训练的transformer模型。我们的分析揭示了几个关键发现:(i)平衡的预训练数据可以在后训练过程中激活潜在能力,(ii)SFT从一组对预训练模型具有挑战性的示例中学习最佳,而过度大的SFT数据集可能会稀释有信息的预训练信号。相比之下,RL在大规模数据上最有效,这些数据对预训练模型来说并不过于困难。我们通过在大型非线性transformer架构上的实验验证了这些理论见解。

英文摘要:

Large Language Models (LLMs) are pretrained on massive datasets and later instruction-tuned via supervised fine-tuning (SFT) or reinforcement learning (RL). Best practices emphasize large, diverse pretraining data, whereas post-training operates differently: SFT relies on smaller, high-quality datasets, while RL benefits more from scale, with larger amounts of feedback often outweighing label quality. Yet it remains unclear why pretraining and RL require large datasets, why SFT excels on smaller ones, and what defines high-quality SFT data. In this work, we theoretically analyze transformers trained on an in-context weight prediction task for linear regression. Our analysis reveals several key findings: $(i)$ balanced pretraining data can induce latent capabilities later activated during post-training, and $(ii)$ SFT learns best from a small set of examples challenging for the pretrained model, while excessively large SFT datasets may dilute informative pretraining signals. In contrast, RL is most effective on large-scale data that is not overly difficult for the pretrained model. We validate these theoretical insights with experiments on large nonlinear transformer architectures.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑