临界初始化在宽标量输入网络中破坏高阶输入导数
Critical initialization destabilizes higher input derivatives in wide scalar-input networks
浏览论文内容
中文总结 AI 辅助
本文研究宽标量输入网络在临界初始化下高阶输入导数的行为，推导出三阶均场递推并证明二阶导数方差线性增长，残差网络导数方差有界，模拟验证了理论结果。
中文摘要 AI 辅助
混沌边缘条件在宽随机初始化网络中保持一阶输入扰动，但物理信息损失、分数匹配和导数正则化依赖于更高阶的输入导数。对于光滑的标量输入全连接网络，利用在无限宽度极限下每个固定深度处成立的有限导数射流的联合高斯性，我们推导出精确到三阶的均场递推关系，这些关系在方差不动点处是精确的，并具有几何衰减的有限深度修正。在临界状态下，一阶导数方差是深度不变的，而每当激活函数具有非零曲率时，二阶导数方差线性增长。由此产生的三阶系统在均场磁化率上闭合。对于分支尺度为L^{-1/2}的残差网络，我们证明在明确的规则性假设下，每个固定的有限导数阶具有一致有界的方差。模拟验证了临界增长规律、残差界和闭合递推关系。这些结果涉及初始化，而非训练后网络的性能。
英文摘要
The edge-of-chaos condition preserves first-order input perturbations in wide randomly initialized networks, but physics-informed losses, score matching and derivative regularization depend on higher input derivatives. For smooth scalar-input fully connected networks, using a joint Gaussianity of the finite derivative jet that holds in the infinite-width limit at each fixed depth, we derive mean-field recursions through third order that are exact at the variance fixed point, with finite-depth corrections that decay geometrically. At criticality, the first-derivative variance is depth-invariant, whereas the second-derivative variance grows linearly whenever the activation has nonzero curvature. The resulting third-order system closes on mean-field susceptibilities. For residual networks with branch scale L^{-1/2}, we prove that every fixed finite derivative order has uniformly bounded variance under explicit regularity assumptions. Simulations verify the critical growth laws, the residual bound, and the closed recursion. The results concern initialization, not trained-network performance.
发表机构
- Indian Institute of Technology Ropar（印度理工学院鲁尔基分校）
机构由 AI 辅助整理，请以论文原文为准。
