arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.18534cs.LG
收藏

通过扩展合成巨文档实现数据高效的预训练

Data-efficient pre-training by scaling synthetic megadocs

Stanford University(斯坦福大学)

Konwoo Kim, Suhas Kotha, Yejin Choi, Tatsunori Hashimoto, Nick Haber, Percy Liang

AI总结:

本文研究如何通过生成更长的合成巨文档提升预训练效果,发现其在降低损失和提升基准准确率方面优于简单重述,数据效率从1.48倍提升至1.80倍。

摘要:

合成数据增强已成为在数据受限而非计算受限的情况下预训练的有希望的解决方案。我们研究如何设计合成数据算法以实现更好的损失扩展:不仅在有限计算下降低损失,而且当计算趋于无穷时也如此。我们首先表明,在网页数据中混合合成生成的重述可以改善i.i.d.验证损失,尽管合成数据来自完全不同的分布。通过最优混合和轮数,损失和基准准确率在合成生成数量增加时提高,不发生过拟合,最终在32次重述每文档时达到1.48倍的数据效率。我们发现从新视角下,同一文档的合成生成可以形成单一显著更长的巨文档而非多个短文档。我们展示了两种构建巨文档的方法:将同一网页文档的合成重述拼接在一起或通过插入推理扩展文档。这两种方法相对于简单重述在i.i.d.损失、下游基准和尤其长上下文损失方面均有提升,数据效率从1.48倍提升至1.80倍在32次生成每文档时。重要的是,巨文档相对于简单重述的改进随着更多合成数据的生成而扩大。我们的结果展示了如何设计受益于增加计算量的合成数据算法,当数据受限时。

英文摘要:

Synthetic data augmentation has emerged as a promising solution when pre-training is constrained by data rather than compute. We study how to design synthetic data algorithms that achieve better loss scaling: not only lowering loss at finite compute but especially as compute approaches infinity. We first show that pre-training on web data mixed with synthetically generated rephrases improves i.i.d. validation loss on the web data, despite the synthetic data coming from an entirely different distribution. With optimal mixing and epoching, loss and benchmark accuracy improve without overfitting as the number of synthetic generations grows, plateauing near $1.48\times$ data efficiency at 32 rephrases per document. We find even better loss scaling under a new perspective: synthetic generations from the same document can form a single substantially longer megadocument instead of many short documents. We show two ways to construct megadocs: stitching synthetic rephrases from the same web document or stretching a document by inserting rationales. Both methods improve i.i.d. loss, downstream benchmarks, and especially long-context loss relative to simple rephrasing, increasing data efficiency from $1.48\times$ to $1.80\times$ at $32$ generations per document. Importantly, the improvement of megadocs over simple rephrasing widens as more synthetic data is generated. Our results show how to design synthetic data algorithms that benefit more from increasing compute when data-constrained.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑