arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2306.14975cs.LGcond-mat.dis-nnhep-thmath.PRstat.ML
收藏

复杂数据集的底层标度律与普适统计结构

The Underlying Scaling Laws and Universal Statistical Structure of Complex Datasets

École Polytechnique Fédérale de Lausanne (EPFL)(洛桑联邦理工学院)

Noam Levi, Yaron Oz

AI总结:

该研究借助统计物理与随机矩阵理论分析特征协方差矩阵,揭示真实与生成数据集的普适幂律标度和混沌普适类,并证明自然图像Gram矩阵可由简单Wishart矩阵近似,为神经网络研究提供理论基础。

摘要:

我们研究在真实世界复杂数据集以及人工生成数据集中涌现的普适特征。我们的方法是将数据类比为物理系统,并运用统计物理学与随机矩阵理论(Random Matrix Theory, RMT)的工具来揭示其底层结构。我们聚焦于特征-特征协方差矩阵,分析其局部与全局特征值统计。主要观察结果如下:(i)对于不相关的正态分布数据,其特征值主体所表现出的幂律标度与真实世界数据相比存在巨大差异;(ii)这种标度行为可以通过生成长程相关的高斯数据来完全建模;(iii)从RMT的角度看,生成数据集与真实世界数据集处于同一普适类中,表现为混沌系统而非可积系统;(iv)预期的RMT统计行为已经在经验协方差矩阵上显现,且所需的数据集规模显著小于传统真实世界训练所用的规模,并且可以与逼近总体幂律标度行为所需的样本数相关联;(v)Shannon熵与局部RMT结构及特征值标度相关,在强相关数据集中显著小于不相关数据集,并且需要更少的样本即可达到分布熵。这些发现表明,在样本量充足的条件下,自然图像数据集的Gram矩阵可以很好地用一个具有简单协方差结构的Wishart随机矩阵来近似,这为依赖数据Gram矩阵的神经网络动力学与泛化能力的严格研究打开了大门。

英文摘要:

We study universal traits which emerge both in real-world complex datasets, as well as in artificially generated ones. Our approach is to analogize data to a physical system and employ tools from statistical physics and Random Matrix Theory (RMT) to reveal their underlying structure. We focus on the feature-feature covariance matrix, analyzing both its local and global eigenvalue statistics. Our main observations are: (i) The power-law scalings that the bulk of its eigenvalues exhibit are vastly different for uncorrelated normally distributed data compared to real-world data, (ii) this scaling behavior can be completely modeled by generating Gaussian data with long range correlations, (iii) both generated and real-world datasets lie in the same universality class from the RMT perspective, as chaotic rather than integrable systems, (iv) the expected RMT statistical behavior already manifests for empirical covariance matrices at dataset sizes significantly smaller than those conventionally used for real-world training, and can be related to the number of samples required to approximate the population power-law scaling behavior, (v) the Shannon entropy is correlated with local RMT structure and eigenvalues scaling, is substantially smaller in strongly correlated datasets compared to uncorrelated ones, and requires fewer samples to reach the distribution entropy. These findings show that with sufficient sample size, the Gram matrix of natural image datasets can be well approximated by a Wishart random matrix with a simple covariance structure, opening the door to rigorous studies of neural network dynamics and generalization which rely on the data Gram matrix.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑