arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2412.14872cs.CL
收藏

自回归语言模型在真实世界数据为有限集时发生坍缩的理论证明

Theoretical Proof that Auto-regressive Language Models Collapse when Real-world Data is a Finite Set

Key Laboratory of High Confidence Software Technologies (Peking University), Ministry of Education(高可信软件技术重点实验室(北京大学));School of Computer Science, Peking University(北京大学计算机学院);University of Wisconsin-Madison(威斯康星大学麦迪逊分校)

Lecheng Wang, Xianjie Shi, Ge Li, Jia Li, Xuanming Zhang, Yihong Dong, Wenpin Jiao, Hong Mei

AI总结:

该研究针对自回归语言模型递归训练坍缩问题,通过理论证明得出有限真实数据语料库加入生成数据后模型坍缩必然发生的结论,指出限制合成数据量无法根本缓解坍缩,关键在于提升合成数据质量。

摘要:

自回归语言模型(LM)已被广泛用于在数据稀缺领域生成数据以训练新的语言模型,弥补真实世界数据的不足。此前的研究通过实验发现,当语言模型在递归生成的数据上训练时会发生坍缩。本文提出了一项理论证明:一旦某个语料库(例如万维网的一个子集)开始纳入生成的数据,且没有新的真实世界数据被加入该语料库,那么无论每个语言模型生成并贡献给语料库的数据量有多小,经过足够长的时间后,语言模型坍缩都不可避免。这一发现表明,试图通过限制语料库中合成数据的数量来缓解坍缩的做法从根本上是不足的。相反,避免坍缩的关键在于确保合成数据的质量。

英文摘要:

Auto-regressive language models (LMs) have been widely used to generate data in data-scarce domains to train new LMs, compensating for the scarcity of real-world data. Previous work experimentally found that LMs collapse when trained on recursively generated data. This paper presents a theoretical proof: once a corpus (such as a subset of the World Wide Web) begins to incorporate generated data and no new real-world data is added to the corpus, then no matter how small the amount of data each LM generates and contributes to the corpus, LM collapse is inevitable after sufficient time. This finding suggests that attempts to mitigate collapse by limiting the quantity of synthetic data in the corpus are fundamentally insufficient. Instead, avoiding collapse hinges on ensuring the quality of synthetic data.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑