MimicNorm：权重均值与最后一个BN层模拟批量归一化的动态特性
MimicNorm: Weight Mean and Last BN Layer Mimic the Dynamic of Batch Normalization
AI总结：
针对批量归一化（BN）内存开销大、微批次下不准确的问题，提出仅含权重均值操作和末层BN的轻量归一化方法MimicNorm，经NTK理论验证和实验证明，可在保持准确率的同时降低约20%内存消耗。
摘要：
大量实验已证实批量归一化（Batch Normalization，BN）层在提升收敛性和泛化能力方面的成效。然而，BN需要额外的内存和浮点计算。此外，由于BN依赖批次统计量，在微批次下其计算结果会不准确。本文中，我们通过简化BN的正则化机制，同时保留BN层的两个核心作用——即数据去相关和自适应学习率——来解决这些问题。我们提出了一种新型归一化方法，名为MimicNorm，用于提升网络训练的收敛性和效率。MimicNorm仅包含两个轻量操作，包括改进的权重均值操作（从权重参数张量中减去均值）以及损失函数前的一个BN层（最后一个BN层）。我们利用神经切线核（neural tangent kernel，NTK）理论证明，我们的权重均值操作能够白化激活值，并像BN层一样将网络过渡到混沌状态，从而提升收敛性能。最后一个BN层可提供自动调整的学习率，同时还能提高准确率。实验结果表明，在包括ResNets以及ShuffleNet等轻量级网络在内的多种网络结构上，MimicNorm取得了相近的准确率，同时内存消耗降低了约20%。代码已公开在https://github.com/Kid-key/MimicNorm。
英文摘要：
Substantial experiments have validated the success of Batch Normalization (BN) Layer in benefiting convergence and generalization. However, BN requires extra memory and float-point calculation. Moreover, BN would be inaccurate on micro-batch, as it depends on batch statistics. In this paper, we address these problems by simplifying BN regularization while keeping two fundamental impacts of BN layers, i.e., data decorrelation and adaptive learning rate. We propose a novel normalization method, named MimicNorm, to improve the convergence and efficiency in network training. MimicNorm consists of only two light operations, including modified weight mean operations (subtract mean values from weight parameter tensor) and one BN layer before loss function (last BN layer). We leverage the neural tangent kernel (NTK) theory to prove that our weight mean operation whitens activations and transits network into the chaotic regime like BN layer, and consequently, leads to an enhanced convergence. The last BN layer provides autotuned learning rates and also improves accuracy. Experimental results show that MimicNorm achieves similar accuracy for various network structures, including ResNets and lightweight networks like ShuffleNet, with a reduction of about 20% memory consumption. The code is publicly available at https://github.com/Kid-key/MimicNorm.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
