发表机构
School of Mathematical Sciences, Shanghai Jiao Tong University; Institute of Natural Sciences, MOE-LSC, Shanghai Jiao Tong University（上海交通大学数学科学学院; 上海交通大学自然科学研究院）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本文通过三阶段动态分析揭示权重衰减在两层ReLU网络中促进神经元凝聚的机制，超越了单纯减少参数范数的传统理解。
AI 中文摘要
权重衰减被广泛用作神经网络训练中的正则化技术，但它在神经元凝聚（参数方向对齐）中的作用仍不清楚。从神经正切核区域的参数初始化出发，我们通过三个阶段刻画了权重衰减下的训练动态：快速拟合、幅度压缩和神经元凝聚。使用两层ReLU网络，我们分析了一个控制神经元幅度和方向的残差相关场。在快速拟合期间，残差接近由权重衰减维持的准静态平衡，而神经正切核几乎保持不变。在幅度压缩的早期阶段，核衰减放大了残差相关场，其孤立的吸引极值提供了凝聚方向的候选。随着神经元幅度的稳定，我们限制了吸引极值的漂移，并证明了神经元方向围绕它们的收缩，从而导致神经元凝聚。这种阶段性分析提供了对权重衰减如何促进凝聚表示（超越减少参数范数）的动态理解。
英文摘要
Weight decay is widely used as a regularization technique in neural network training, yet its role in neuron condensation (parameter direction alignment) remains unclear. Starting from a parameter initialization in the neural tangent kernel regime, we characterize training dynamics under weight decay through three stages: rapid fitting, amplitude compression, and neuron condensation. Using a two-layer ReLU network, we analyze a residual correlation field that governs both neuron amplitudes and directions. During rapid fitting, the residual approaches a quasi-static equilibrium maintained by weight decay while the tangent kernel remains nearly unchanged. In the early stage of amplitude compression, kernel decay amplifies the residual correlation field, whose isolated attracting extrema provide candidates for condensation directions. As neuron amplitudes stabilize, we bound the drift of attracting extrema and demonstrate contraction of neuron directions around them, leading to neuron condensation. This staged analysis provides a dynamical understanding of how weight decay promotes a condensed representation, beyond reducing parameter norms.
Comments34 pages, 8 figures
