训练轻度参数化神经网络的早期收敛性与全局收敛性
Early Stage Convergence and Global Convergence of Training Mildly Parameterized Neural Networks
AI总结：
本文针对轻度参数化神经网络，通过神经元激活模式的微观分析推导更强的梯度下界，证明了GD和SGD在随机初始化下的早期快速收敛性，并在指数型损失下进一步证明了GD的全局收敛性。
摘要：
本文研究了从随机初始化开始训练轻度参数化神经网络时梯度下降（GD）和随机梯度下降（SGD）的收敛性。对于广泛的模型和损失函数，包括最常用的平方损失和交叉熵损失，我们证明了一个“早期收敛”结果。我们表明，在训练的早期阶段，损失会显著下降，并且这种下降是快速的。此外，对于指数型损失函数，并在训练数据满足一定假设的条件下，我们证明了梯度下降的全局收敛性。我们的研究不依赖于极端的过度参数化，而是基于对神经元激活模式的微观分析，这帮助我们推导出更强的梯度下界。这些关于激活模式的结果，我们称之为“神经元划分”，有助于建立理解神经网络训练动力学行为的直觉，并可能具有独立的研究价值。
英文摘要：
The convergence of GD and SGD when training mildly parameterized neural networks starting from random initialization is studied. For a broad range of models and loss functions, including the most commonly used square loss and cross entropy loss, we prove an ``early stage convergence'' result. We show that the loss is decreased by a significant amount in the early stage of the training, and this decrease is fast. Furthurmore, for exponential type loss functions, and under some assumptions on the training data, we show global convergence of GD. Instead of relying on extreme over-parameterization, our study is based on a microscopic analysis of the activation patterns for the neurons, which helps us derive more powerful lower bounds for the gradient. The results on activation patterns, which we call ``neuron partition'', help build intuitions for understanding the behavior of neural networks' training dynamics, and may be of independent interest.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
