arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.04840cs.LGstat.ML
收藏

强模型坍缩

Strong Model Collapse

FAIR, Meta(FAIR(Meta));NYU Center for Data Science(纽约大学数据科学中心);NYU Courant Institute(纽约大学柯朗数学科学研究所);UCLA(加利福尼亚大学洛杉矶分校)

Elvis Dohmatob, Yunzhen Feng, Arjun Subramonian, Julia Kempe

AI总结:

该研究在缩放定律范式下证实强模型坍缩现象存在,发现极低占比的合成数据即可引发该问题,且大模型通常会放大坍缩,仅在极高插值阈值以上略有缓解,相关结论经语言模型与图像前馈网络实验验证。

摘要:

在支撑ChatGPT、Llama等大型神经网络训练的scaling laws(缩放定律)范式下,我们针对有监督回归场景,证实了强形式模型坍缩现象的存在——这是训练语料中包含合成数据引发的严重性能退化问题。研究结果显示,即便合成数据占比极低(例如仅占总训练数据集的1%),仍会导致模型坍缩:不断扩大训练集规模也无法提升性能。我们进一步探究了增大模型规模这一符合当前大语言模型训练趋势的方法,会加剧还是缓解模型坍缩。在通过可调规模的随机投影近似神经网络的简化场景中,我们从理论和实证两方面证明,更大的模型会放大模型坍缩效应。有趣的是,我们的理论还表明,在插值阈值(对于超大规模数据集而言该阈值可能极高)之上,更大的模型或许能缓解坍缩,但无法完全避免该问题。我们的理论发现已通过在语言模型和处理图像的前馈神经网络上开展的实验得到实证验证。

英文摘要:

Within the scaling laws paradigm, which underpins the training of large neural networks like ChatGPT and Llama, we consider a supervised regression setting and establish the existance of a strong form of the model collapse phenomenon, a critical performance degradation due to synthetic data in the training corpus. Our results show that even the smallest fraction of synthetic data (e.g., as little as 1\% of the total training dataset) can still lead to model collapse: larger and larger training sets do not enhance performance. We further investigate whether increasing model size, an approach aligned with current trends in training large language models, exacerbates or mitigates model collapse. In a simplified regime where neural networks are approximated via random projections of tunable size, we both theoretically and empirically show that larger models can amplify model collapse. Interestingly, our theory also indicates that, beyond the interpolation threshold (which can be extremely high for very large datasets), larger models may mitigate the collapse, although they do not entirely prevent it. Our theoretical findings are empirically verified through experiments on language models and feed-forward neural networks for images.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑