arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.01255cs.LGcs.AI
收藏

关于层归一化的非线性

On the Nonlinearity of Layer Normalization

Beihang University(北京航空航天大学);Institute of Artificial Intelligence(人工智能研究院)

Yunhao Ni, Yuxin Guo, Junlong Jia, Lei Huang

AI总结:

本文研究了层归一化(LN)的非线性与表示能力,证明仅含少量神经元和LN层的网络可分类任意标签样本,并揭示通过分组划分放大LN非线性以指导神经架构设计的有效性。

摘要:

层归一化(LN)是深度学习中普遍使用的技术,但对其的理论理解仍然难以捉摸。本文研究了LN的一个新理论方向,即其非线性与表示能力。我们研究了由线性和LN变换逐层组合构成的网络(称为LN-Net)的表示能力。我们从理论上证明,给定$m$个具有任意标签分配的样本,一个每层仅有3个神经元且具有$O(m)$个LN层的LN-Net能够正确分类它们。我们进一步展示了LN-Net的VC维度的下界。LN的非线性可以通过分组划分来放大,这在温和假设下得到了理论证明,并得到了我们实验的实证支持。基于我们的分析,我们考虑通过利用和放大LN的非线性来设计神经架构,其有效性得到了我们实验的支持。

英文摘要:

Layer normalization (LN) is a ubiquitous technique in deep learning but our theoretical understanding to it remains elusive. This paper investigates a new theoretical direction for LN, regarding to its nonlinearity and representation capacity. We investigate the representation capacity of a network with layerwise composition of linear and LN transformations, referred to as LN-Net. We theoretically show that, given $m$ samples with any label assignment, an LN-Net with only 3 neurons in each layer and $O(m)$ LN layers can correctly classify them. We further show the lower bound of the VC dimension of an LN-Net. The nonlinearity of LN can be amplified by group partition, which is also theoretically demonstrated with mild assumption and empirically supported by our experiments. Based on our analyses, we consider to design neural architecture by exploiting and amplifying the nonlinearity of LN, and the effectiveness is supported by our experiments.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑