长尾的故事：作为缩放定律变化的模型崩溃
A Tale of Tails: Model Collapse as a Change of Scaling Laws
AI总结：
本文研究合成数据进入训练语料库后缩放定律的变化，通过理论框架分析模型崩溃现象，并在算术任务和文本生成的大规模实验中验证了理论。
摘要：
随着AI模型规模的增长，神经缩放定律已成为预测在增加容量和原始（人类或自然）训练数据规模时大型模型改进的重要工具。然而，流行模型的广泛应用意味着在线数据和文本的生态系统将共同演化，逐渐包含越来越多的合成数据。在本文中，我们提出问题：在合成数据不可避免地进入训练语料库的情况下，缩放定律将如何改变？未来的模型仍会改进，还是注定会退化直至完全（模型）崩溃？我们通过缩放定律的视角开发了模型崩溃的理论框架。我们发现了广泛的退化现象，分析了缩放的丧失、随代数变化的偏移缩放、技能的“遗忘”，以及混合人类和合成数据时的顿悟。我们的理论通过在算术任务上使用transformer以及使用大型语言模型Llama2进行文本生成的大规模实验得到了验证。
英文摘要：
As AI model size grows, neural scaling laws have become a crucial tool to predict the improvements of large models when increasing capacity and the size of original (human or natural) training data. Yet, the widespread use of popular models means that the ecosystem of online data and text will co-evolve to progressively contain increased amounts of synthesized data. In this paper we ask: How will the scaling laws change in the inevitable regime where synthetic data makes its way into the training corpus? Will future models, still improve, or be doomed to degenerate up to total (model) collapse? We develop a theoretical framework of model collapse through the lens of scaling laws. We discover a wide range of decay phenomena, analyzing loss of scaling, shifted scaling with number of generations, the ''un-learning" of skills, and grokking when mixing human and synthesized data. Our theory is validated by large-scale experiments with a transformer on an arithmetic task and text generation using the large language model Llama2.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
