小组白化:平衡学习效率和代表能力
Group Whitening: Balancing Learning Efficiency and Representational Capacity
摘要:
批量归一化(BN)是一种重要的技术,通常被结合到深度学习模型中以在小批量内执行标准化。BN在提高模型学习效率方面的优点可以通过应用白化来进一步放大,而其在估计用于推断的总体统计量方面的缺点可以通过组归一化(GN)来避免。本文提出了组白化(GW),它利用白化操作的优点,并避免了小批量归一化的缺点。此外,我们分析了规范化对特征的约束,并从模型的代表能力的角度展示了批量大小(组数)如何影响批量(组)规范化网络的性能。这一分析为GW在实践中的应用提供了理论指导。最后,我们将所提出的GW应用于ResNet和ResNeXt架构,并在ImageNet和COCO基准上进行实验。结果表明,GW一致地提高了不同架构的性能,在ImageNet上,$1.02\%$ $\sim$ $1.49\%$在top-1准确度中的绝对增益,以及在COCO上,限界框AP中的$1.82\%$ $\sim$ $3.21\%$的绝对增益。
英文摘要:
Batch normalization (BN) is an important technique commonly incorporated into deep learning models to perform standardization within mini-batches. The merits of BN in improving a model's learning efficiency can be further amplified by applying whitening, while its drawbacks in estimating population statistics for inference can be avoided through group normalization (GN). This paper proposes group whitening (GW), which exploits the advantages of the whitening operation and avoids the disadvantages of normalization within mini-batches. In addition, we analyze the constraints imposed on features by normalization, and show how the batch size (group number) affects the performance of batch (group) normalized networks, from the perspective of model's representational capacity. This analysis provides theoretical guidance for applying GW in practice. Finally, we apply the proposed GW to ResNet and ResNeXt architectures and conduct experiments on the ImageNet and COCO benchmarks. Results show that GW consistently improves the performance of different architectures, with absolute gains of $1.02\%$ $\sim$ $1.49\%$ in top-1 accuracy on ImageNet and $1.82\%$ $\sim$ $3.21\%$ in bounding box AP on COCO.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。