关于激活和归一化在初始化时获得等距嵌入的影响
On the impact of activation and normalization in obtaining isometric embeddings at initialization
AI总结：
本文研究深度神经网络初始化时倒数第二层 Gram 矩阵的退化问题，证明层归一化结合非线性激活能以指数速率将该矩阵偏置向单位矩阵，并利用 Hermite 展开式量化了该速率。
摘要：
本文探讨了深度神经网络中倒数第二层 Gram 矩阵的结构，该矩阵包含一批输入对应输出的成对内积。在多种架构中观察到，该 Gram 矩阵在初始化时随深度退化，从而显著减慢训练。批归一化或层归一化等归一化层在防止秩塌缩问题中起关键作用。尽管已有进展，现有理论结果无法推广到 Transformer 中广泛使用的层归一化，也无法定量刻画非线性激活的作用。为弥补这一空白，我们证明层归一化结合激活层，在初始化时以随深度指数增长的速率将多层感知机的 Gram 矩阵偏置向单位矩阵。我们利用激活函数的 Hermite 展开式量化了该速率。
英文摘要：
In this paper, we explore the structure of the penultimate Gram matrix in deep neural networks, which contains the pairwise inner products of outputs corresponding to a batch of inputs. In several architectures it has been observed that this Gram matrix becomes degenerate with depth at initialization, which dramatically slows training. Normalization layers, such as batch or layer normalization, play a pivotal role in preventing the rank collapse issue. Despite promising advances, the existing theoretical results do not extend to layer normalization, which is widely used in transformers, and can not quantitatively characterize the role of non-linear activations. To bridge this gap, we prove that layer normalization, in conjunction with activation layers, biases the Gram matrix of a multilayer perceptron towards the identity matrix at an exponential rate with depth at initialization. We quantify this rate using the Hermite expansion of the activation function.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
