关联流支配临界状态下的学习过程
Correlation flow governs learning at criticality
- CNRS（法国国家科学研究中心）
- ENS de Lyon（里昂高等师范学院）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
该研究结合平均场与随机矩阵理论，揭示关联传播与NTK的联系，证明临界状态下正交初始化可抑制有限尺寸修正，其理论在有限尺寸网络上得到验证，阐明了深度学习渐近动力学的关键机制。
AI中文摘要：
深度神经网络的初始化决定了信息和梯度能否跨深度传播，但将这些特性与学习动力学关联起来的统一理论仍未形成。结合平均场理论与随机矩阵理论，我们建立了关联传播与神经正切核（Neural Tangent Kernel, NTK）之间的直接联系，NTK支配着无限宽、无限深网络的序列极限学习过程。关联向无限深度的传播仅在权重-偏置方差平面内的单个临界点处可行。在该临界点，我们证明端到端雅可比矩阵随深度呈代数级消失，并以此推导出NTK在无限深度下恰好与输出关联成正比。信息传播与学习动力学间的这种等价关系此前未被发现。我们进一步证明，正交初始化可抑制高斯初始化下存在的主导有限尺寸修正，阐明了两种初始化集合在该极限下的各自作用。这些理论预测在有限宽度、有限深度的网络上得到了定量验证。综上，这些结果表明，临界状态下的正交初始化在控制深度学习的渐近动力学中发挥核心作用。
英文摘要：
The initialization of deep neural networks determines whether information and gradients can propagate across depth, yet a unified theory connecting these properties to learning dynamics remains elusive. Combining mean-field theory and random matrix theory, we establish a direct link between correlation propagation and the Neural Tangent Kernel (NTK) that governs learning in the sequential limit of infinitely wide, infinitely deep networks. Correlation propagation to infinite depth is possible only at a single, critical point in the weight-bias variance plane. At this point, we leverage the algebraic decay of the end-to-end Jacobian with depth to prove that the NTK becomes exactly proportional to the output correlation at infinite depth, tying together information propagation and learning dynamics. We further show that orthogonal initialization suppresses the leading finite-size corrections present under Gaussian initialization, clarifying the respective roles of the two initialization ensembles in this limit. These theoretical predictions are validated quantitatively on finite-width, finite-depth networks. Together, these results demonstrate that orthogonal initialization and criticality are required to control the asymptotic dynamics of deep learning.
