LayerNorm and RMSNorm impose fundamentally different geometric constraints on their outputs - and this difference has a precise, quantifiable consequence for model complexity. We prove that LayerNorm's mean-centering step, by confining data to a linear hyperplane (through the origin), reduces the Local Learning Coefficient (LLC) of the subsequent weight matrix by exactly $m/2$ (where $m$ is its output dimension); RMSNorm's projection onto a sphere preserves the LLC entirely. This reduction is structurally guaranteed before any training begins, determined by data manifold geometry alone.
The underlying condition is a geometric threshold: for the codimension-one manifolds we study, the LLC drop is binary -- any non-zero curvature, regardless of sign or magnitude, is sufficient to preserve the LLC, while only affinely flat manifolds cause the drop. At finite sample sizes this threshold acquires a smooth crossover whose width depends on how much of the data distribution actually experiences the curvature, not merely on whether curvature exists somewhere.
We verify both predictions experimentally with controlled single-layer scaling experiments using the wrLLC framework. We further show that Softmax simplex data introduces a "smuggled bias" that activates the same $m/2$ LLC drop when paired with an explicit downstream bias, proved via the affine symmetry extension of the main theorem and confirmed empirically.
Training-Time Batch Normalization Reshapes Local Partition Geometry in Piecewise-Affine Networks
AI for Good Istituto Italiano di Tecnologia(AI for Good 意大利技术研究院);DITEN University of Genoa(DITEN Genoa大学);State Key Laboratory of Novel Software Technology School of Intelligence Science and Technology(新型软件技术国家重点实验室 智能科学与技术学院);School of Artificial Intelligence Nanjing University(人工智能学院 南京大学);Department of Computer Science University of Verona(计算机科学系 Verona大学)
Enhancing Reliability of Neural Networks at the Edge: Inverted Normalization with Stochastic Affine Transformations
Karlsruhe Institute of Technology(卡尔斯鲁厄理工学院);Univ. Grenoble Alpes, CEA, CNRS, Grenoble INP, and IRIG-Spintec(格勒诺布尔阿尔卑斯大学、法国原子能委员会、法国国家科学研究中心、格勒诺布尔国立理工学院及IRIG-Spintec)
Uncertainty Estimations by Softplus normalization in Bayesian Convolutional Neural Networks with Variational Inference
TU Kaiserslautern(凯泽斯劳滕工业大学);MindGarage;Imperial College London(帝国理工学院);NeuralSpace;Luleå University of Technology(吕勒奥理工大学)
我们介绍了一种新的不确定性估计贝叶斯卷积神经网络与变分推理的分类任务。通过在最后一层对Softplus函数的输出进行归一化,我们以连贯的方式估计任意和认知的不确定性。权重的后验概率分布由Bayes by Backprop推断。首先,我们展示了这种可靠的变分推理方法可以作为各种网络架构的基本构造。在监督学习设置(MNIST,CIFAR-10,CIFAR-100)的多个数据集上,这种变分推理方法在相同的体系结构中实现了与频率论推理相当的性能,而两个必要的,不确定性和正则化的度量自然地结合在一起。其次,我们研究了我们提出的任意性和认知不确定性的度量是如何推导出来的,并在上述数据集上进行了验证。
Parallel Layer Normalization for Universal Approximation
State Key Laboratory of Complex and Critical Software Environment (SKLCCSE)(复杂与关键软件环境国家重点实验室);Beihang University(北京航空航天大学);Hangzhou International Innovation Institute(杭州国际创新研究院)
Phasor Attention: Mean Root Square Normalization for Phase Manifold Preservation
College of Pharmacy, Chungnam National University(忠南国立大学药学院);Department of Computer Science & Engineering, Chungnam National University(忠南国立大学计算机科学与工程系)
AffineLens: Capturing the Continuous Piecewise Affine Functions of Neural Networks
State Key Laboratory of Novel Software Technology School of Intelligence Science and Technology Nanjing University, Jiangsu, China(软件新技术国家重点实验室 智能科学与技术学院 南京大学 江苏 中国);AI for Good Istituto Italiano di Tecnologia, Genoa, Italy DITEN University of Genoa, Genoa, Italy(AI for Good 意大利技术研究院,热那亚,意大利 DITEN 热那亚大学,热那亚,意大利);State Key Laboratory of Novel Software Technology School of Artificial Intelligence Nanjing University, Jiangsu, China(软件新技术国家重点实验室 人工智能学院 南京大学 江苏 中国);AI for Good Istituto Italiano di Tecnologia, Genoa, Italy Department of Computer Science University of Verona, Verona, Italy(AI for Good 意大利技术研究院,热那亚,意大利 计算机科学系 热那亚大学,热那亚,意大利)