迈向无深度限制的训练：无梯度爆炸的批量归一化
Towards Training Without Depth Limits: Batch Normalization Without Gradient Explosion
AI总结：
本研究针对批量归一化网络存在的深度方向梯度爆炸问题，构建了带线性激活的批量归一化MLP并基于温加滕微积分证明其任意深度梯度有界，还提出适用于部分非线性激活的激活整形方案，实现信号传播与梯度稳定的兼顾。
摘要：
归一化层是深度神经网络的关键构建模块之一。多项理论研究表明，批量归一化通过避免各层的表征变得共线，改善了信号传播。然而，关于批量归一化的平均场理论研究结果也指出，这种优势是以深度方向上的梯度爆炸为代价的。受批量归一化这两方面特性的启发，本研究提出了如下问题：“批量归一化网络能否保留最优信号传播特性，同时避免梯度爆炸？”我们通过构建一种带有线性激活函数和批量归一化的多层感知器（Multi-Layer Perceptron, MLP），对该问题给出了肯定的答案，该结构可证明在任意深度下都具有有界梯度。基于Weingarten calculus（温加滕微积分），我们为该构建的MLP建立了严格的非渐近理论，精确刻画了前向信号传播过程，同时证明对于线性无关的输入样本（该条件在大多数实际场景中成立），梯度保持有界。受我们的理论启发，我们还设计了一种激活整形方案，实验证明该方案对于某些非线性激活函数也能实现相同的特性。
英文摘要：
Normalization layers are one of the key building blocks for deep neural networks. Several theoretical studies have shown that batch normalization improves the signal propagation, by avoiding the representations from becoming collinear across the layers. However, results on mean-field theory of batch normalization also conclude that this benefit comes at the expense of exploding gradients in depth. Motivated by these two aspects of batch normalization, in this study we pose the following question: "Can a batch-normalized network keep the optimal signal propagation properties, but avoid exploding gradients?" We answer this question in the affirmative by giving a particular construction of an Multi-Layer Perceptron (MLP) with linear activations and batch-normalization that provably has bounded gradients at any depth. Based on Weingarten calculus, we develop a rigorous and non-asymptotic theory for this constructed MLP that gives a precise characterization of forward signal propagation, while proving that gradients remain bounded for linearly independent input samples, which holds in most practical settings. Inspired by our theory, we also design an activation shaping scheme that empirically achieves the same properties for certain non-linear activations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
