arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2601.03385cs.LGmath.PR
收藏

SIGMA:用于LLM模型崩溃的可扩展频谱洞察

SIGMA: Scalable Spectral Insights for LLM Model Collapse

Department of Statistics, University of California, Davis(加州大学戴维斯分校统计学系);Department of Mathematics, Northwestern University(西北大学数学系);Boston University(波士顿大学);Department of Applied Mathematics and Statistics, Johns Hopkins University(约翰霍普金斯大学应用数学与统计学系)

Yi Gu, Lingyou Pang, Xiangkun Ye, Tianyu Wang, Jianyu Lin, Carey E. Priebe, Alexander Aue

AI总结:

本文提出SIGMA框架,通过频谱分析嵌入Gram矩阵来检测LLM模型崩溃,提供数学基础的度量方法,并实现大规模模型的可扩展监测。

摘要:

本文提出SIGMA框架,通过频谱分析嵌入Gram矩阵来检测LLM模型崩溃,提供数学基础的度量方法,并实现大规模模型的可扩展监测。

英文摘要:

The rapid adoption of synthetic data for training Large Language Models (LLMs) has introduced the technical challenge of "model collapse"-a degenerative process where recursive training on model-generated content leads to a contraction of distributional variance and representational quality. While the phenomenology of collapse is increasingly evident, rigorous methods to quantify and predict its onset in high-dimensional spaces remain elusive. In this paper, we introduce SIGMA (Spectral Inequalities for Gram Matrix Analysis), a unified framework that benchmarks model collapse through the spectral lens of the embedding Gram matrix. By deriving and utilizing deterministic and stochastic bounds on the matrix's spectrum, SIGMA provides a mathematically grounded metric to track the contraction of the representation space. Crucially, our stochastic formulation enables scalable estimation of these bounds, making the framework applicable to large-scale foundation models where full eigendecomposition is intractable. We demonstrate that SIGMA effectively captures the transition towards degenerate states, offering both theoretical insights into the mechanics of collapse and a practical, scalable tool for monitoring the health of recursive training pipelines.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑