从尖峰到重尾：揭示神经网络的谱演化
From Spikes to Heavy Tails: Unveiling the Spectral Evolution of Neural Networks
AI总结：
本文旨在弥补深度神经网络中重尾经验谱密度（ESD）出现缺乏理论解释的空白，通过在无梯度噪声设定下分析两层网络，揭示了学习率对ESD重尾形状及泛化能力的影响。
摘要：
现代深度神经网络（NNs）的训练策略倾向于在层权重中诱导出重尾（HT）经验谱密度（ESD）。尽管先前的努力已经表明，HT现象与大型NNs的良好泛化相关，但仍缺乏对其出现原因的理论解释。特别是，理解导致该现象的条件有助于阐明泛化与权重谱之间的相互作用。我们的工作旨在通过提出一个简单且丰富的设定来对HT ESD的出现进行建模，从而弥合这一差距。具体而言，我们提出了一个基于理论的设定，用于在两层NNs的ESD中“打造”重尾，并在没有任何梯度噪声的情况下对HT ESD的出现进行了系统分析。这是首个分析无噪声设定的工作，并且我们还将依赖于优化器（GD/Adam）的（大）学习率纳入了HT ESD分析。我们的结果突出了学习率在训练早期阶段对ESD的Bulk+Spike和HT形状的作用，这可以促进两层NN中的泛化。尽管设定要简单得多，但这些观察结果为大规模NNs的行为提供了启示。
英文摘要：
Training strategies for modern deep neural networks (NNs) tend to induce a heavy-tailed (HT) empirical spectral density (ESD) in the layer weights. While previous efforts have shown that the HT phenomenon correlates with good generalization in large NNs, a theoretical explanation of its occurrence is still lacking. Especially, understanding the conditions which lead to this phenomenon can shed light on the interplay between generalization and weight spectra. Our work aims to bridge this gap by presenting a simple, rich setting to model the emergence of HT ESD. In particular, we present a theory-informed setup for 'crafting' heavy tails in the ESD of two-layer NNs and present a systematic analysis of the HT ESD emergence without any gradient noise. This is the first work to analyze a noise-free setting, and we also incorporate optimizer (GD/Adam) dependent (large) learning rates into the HT ESD analysis. Our results highlight the role of learning rates on the Bulk+Spike and HT shape of the ESDs in the early phase of training, which can facilitate generalization in the two-layer NN. These observations shed light on the behavior of large-scale NNs, albeit in a much simpler setting.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
