arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.27432cs.LGcs.ITmath.IT
收藏

归一化的几何成本:神经网络贝叶斯复杂度的仿射界限

The Geometric Cost of Normalization: Affine Bounds on the Bayesian Complexity of Neural Networks

Universität Bonn(波恩大学)

Sungbae Chun

AI总结:

研究揭示归一化层对模型复杂度的影响,证明归一化操作通过约束数据到线性超平面降低局部学习系数,而球形投影则保持该系数不变,且数据流形几何特性决定此效果。

摘要:

LayerNorm和RMSNorm对输出施加不同的几何约束,这一差异对模型复杂度有可量化的后果。我们证明LayerNorm的均值中心化步骤通过将数据限制到通过原点的线性超平面,使后续权重矩阵的局部学习系数(LLC)减少恰好m/2(其中m是输出维度);RMSNorm对球的投影则完全保持LLC。此减少在训练前结构上得到保证,仅由数据流形几何决定。底层条件是一个几何阈值:对于我们研究的共维一曼fold,LLC下降是二元的——任何非零曲率,无论正负或大小,都足以保持LLC,而只有仿射平坦的流形才会导致下降。在有限样本情况下,此阈值获得平滑的过渡,其宽度取决于数据分布中实际经历曲率的程度,而非仅仅是否存在曲率。我们通过受控的单层缩放实验验证了这两个预测,使用wrLLC框架。此外,我们还展示Softmax简单体数据引入了“隐藏偏差”,当与显式下游偏差配对时,会激活相同的m/2 LLC下降,通过主定理的仿射对称扩展证明,并通过实验证实。

英文摘要:

LayerNorm and RMSNorm impose fundamentally different geometric constraints on their outputs - and this difference has a precise, quantifiable consequence for model complexity. We prove that LayerNorm's mean-centering step, by confining data to a linear hyperplane (through the origin), reduces the Local Learning Coefficient (LLC) of the subsequent weight matrix by exactly $m/2$ (where $m$ is its output dimension); RMSNorm's projection onto a sphere preserves the LLC entirely. This reduction is structurally guaranteed before any training begins, determined by data manifold geometry alone. The underlying condition is a geometric threshold: for the codimension-one manifolds we study, the LLC drop is binary -- any non-zero curvature, regardless of sign or magnitude, is sufficient to preserve the LLC, while only affinely flat manifolds cause the drop. At finite sample sizes this threshold acquires a smooth crossover whose width depends on how much of the data distribution actually experiences the curvature, not merely on whether curvature exists somewhere. We verify both predictions experimentally with controlled single-layer scaling experiments using the wrLLC framework. We further show that Softmax simplex data introduces a "smuggled bias" that activates the same $m/2$ LLC drop when paired with an explicit downstream bias, proved via the affine symmetry extension of the main theorem and confirmed empirically.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑