谱批归一化：频域中的归一化
Spectral Batch Normalization: Normalization in the Frequency Domain
AI总结：
本文提出谱批归一化（SBN），通过在频域对特征图归一化来防止深度网络中特征图爆炸，促进频率分量均匀分布，从而增强正则化效果并提升网络性能。
摘要：
正则化是一系列用于提升深度神经网络泛化能力的技术。在本文中，我们提出了谱批归一化（SBN），这是一种通过在频域（谱域）对特征图进行归一化来提升泛化能力的新型有效方法。在初始化时，没有批归一化（BN）的残差网络的激活值往往会随着网络深度呈指数级爆炸。这导致特征图的范数极大，即使参数相对较小。这种爆炸性动态对学习非常不利。BN 使得缩放因子 $γ, β$ 上的权重衰减正则化，近似等价于对特征图范数的加法惩罚，在一定程度上防止了极大的特征图范数。然而，我们通过实验表明，尽管 BN 有近似的加法惩罚，深度神经网络（DNN）的特征图仍倾向于在网络起始处爆炸，且 DNN 的特征图在整个训练过程中包含极大值。这种现象在非残差网络中也以较弱的形式出现。SBN 通过在频域对特征图进行归一化来解决大特征图问题。在我们的实验中，我们凭经验证明 SBN 防止了初始化时的特征图爆炸和训练期间的大特征图值。此外，频域中特征图的归一化使得频率分量分布更均匀。这阻止了 DNN 依赖特征图的单个频率分量。这些影响连同 SBN 的其他作用，对残差和非残差网络的训练产生了正则化效果。我们通过实验表明，在标准正则化方法之外使用 SBN，显著提升了 DNN 的性能，例如在 ImageNet 上 ResNet50 提升了 0.71%。
英文摘要：
Regularization is a set of techniques that are used to improve the generalization ability of deep neural networks. In this paper, we introduce spectral batch normalization (SBN), a novel effective method to improve generalization by normalizing feature maps in the frequency (spectral) domain. The activations of residual networks without batch normalization (BN) tend to explode exponentially in the depth of the network at initialization. This leads to extremely large feature map norms even though the parameters are relatively small. These explosive dynamics can be very detrimental to learning. BN makes weight decay regularization on the scaling factors $γ, β$ approximately equivalent to an additive penalty on the norm of the feature maps, which prevents extremely large feature map norms to a certain degree. However, we show experimentally that, despite the approximate additive penalty of BN, feature maps in deep neural networks (DNNs) tend to explode at the beginning of the network and that feature maps of DNNs contain large values during the whole training. This phenomenon also occurs in a weakened form in non-residual networks. SBN addresses large feature maps by normalizing them in the frequency domain. In our experiments, we empirically show that SBN prevents exploding feature maps at initialization and large feature map values during the training. Moreover, the normalization of feature maps in the frequency domain leads to more uniform distributed frequency components. This discourages the DNNs to rely on single frequency components of feature maps. These, together with other effects of SBN, have a regularizing effect on the training of residual and non-residual networks. We show experimentally that using SBN in addition to standard regularization methods improves the performance of DNNs by a relevant margin, e.g. ResNet50 on ImageNet by 0.71%.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
