arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2404.05090cs.LGcs.AIcs.CL
收藏

使用合成数据训练到底有多糟?语言模型崩溃的统计分析

How Bad is Training on Synthetic Data? A Statistical Analysis of Language Model Collapse

Technology Innovation Institute(技术创新研究院);NYU Abu Dhabi(纽约大学阿布扎比分校);Simons Institute(西蒙斯研究所);Khalifa University of Science and Technology(哈利法科技大学)

Mohamed El Amine Seddik, Suei-Wen Chen, Soufiane Hayou, Pierre Youssef, Merouane Debbah

AI总结:

本文通过统计模型分析语言模型在递归合成数据训练下的崩溃机制,证明纯合成训练必然崩溃,并给出混合真实数据时可避免崩溃的合成数据上限。

摘要:

model collapse(模型崩溃)现象由 Shumailov 等人于2023年提出,指新模型在由先前训练模型生成的合成数据上训练时出现的性能退化。这种递归训练循环会使原始分布的尾部消失,从而使后代模型遗忘初始(真实)分布。为严格理解语言模型中的模型崩溃,本文考虑了一个统计模型,使我们能够刻画各种递归训练场景的影响。具体而言,我们证明仅在合成数据上训练时无法避免模型崩溃。然而,在混合真实数据与合成数据时,我们给出了合成数据最大数量的估计,低于该数量时最终可以避免模型崩溃。我们的理论结论进一步得到了实证验证的支持。

英文摘要:

The phenomenon of model collapse, introduced in (Shumailov et al., 2023), refers to the deterioration in performance that occurs when new models are trained on synthetic data generated from previously trained models. This recursive training loop makes the tails of the original distribution disappear, thereby making future-generation models forget about the initial (real) distribution. With the aim of rigorously understanding model collapse in language models, we consider in this paper a statistical model that allows us to characterize the impact of various recursive training scenarios. Specifically, we demonstrate that model collapse cannot be avoided when training solely on synthetic data. However, when mixing both real and synthetic data, we provide an estimate of a maximal amount of synthetic data below which model collapse can eventually be avoided. Our theoretical conclusions are further supported by empirical validations.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑