具有幂律分布核特征值的随机特征回归中神经缩放定律的动力学
Dynamics of neural scaling laws in random feature regression with powerlaw-distributed kernel eigenvalues
AI总结：
通过推导动力学平均场理论，统一了多种学习机制，揭示了高维神经网络泛化误差的典型行为，并定量解释了神经缩放定律和早停效应。
摘要：
训练大型神经网络揭示了泛化误差的神经缩放定律，这指向高维学习中跨网络架构的普遍行为。还表明这种效应在高度过参数化网络的极限以及神经网络高斯过程极限中持续存在。我们在此对神经网络高斯过程回归动力学中泛化的典型行为建立了原理性理解。我们推导了一个动力学平均场理论，捕捉了典型情况下的学习动力学：这使我们能够统一当前文献中研究的多个现有学习机制，即高斯过程的贝叶斯推断、带或不带权重衰减的梯度流以及随机朗之万训练动力学。利用统计物理学的工具，我们在这些情况下推导的统一框架提供了网络动力学的高维微观行为在低维序参数方面的有效描述。我们表明，集体训练动力学可以分解为N个独立本征模的动力学，这些演化方程仅通过集体响应函数和有效独立噪声的共同统计量耦合。我们的方法通过将学习在具有幂律谱的数据上的谱和动力学特性联系起来，定量解释了泛化误差的动力学，包括神经缩放定律和早停效应等现象。
英文摘要：
Training large neural networks exposes neural scaling laws for the generalization error, which points to a universal behavior across network architectures of learning in high dimensions. It was also shown that this effect persists in the limit of highly overparametrized networks as well as the Neural network Gaussian process limit. We here develop a principled understanding of the typical behavior of generalization in Neural Network Gaussian process regression dynamics. We derive a dynamical mean-field theory that captures the typical case learning dynamics: This allows us to unify multiple existing regimes of learning studied in the current literature, namely Bayesian inference on Gaussian processes, gradient flow with or without weight-decay, and stochastic Langevin training dynamics. Employing tools from statistical physics, the unified framework we derive in either of these cases yields an effective description of the high-dimensional microscopic behavior of networks dynamics in terms of lower dimensional order parameters. We show that collective training dynamics may be separated into the dynamics of N independent eigenmodes, those evolution equations are only coupled through collective response functions and a common statistics of an effective, independent noise. Our approach allows us to quantitatively explain the dynamics of the generalization error by linking spectral and dynamical properties of learning on data with power law spectra, including phenomena such as neural scaling laws and the effect of early stopping.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
