基于层输出单位化的深度神经网络内协变量偏移定界算法
An Internal Covariate Shift Bounding Algorithm for Deep Neural Networks by Unitizing Layers' Outputs
摘要：
已经提出了批量归一化（BN）技术以通过尝试保持层输出的分布不变来减少所谓的内部协变量偏移（ICS）。实验证明了它们在训练深度神经网络方面的有效性。然而，由于在这些BN技术中仅控制前两个时刻，因此似乎对层分布施加了弱约束，并且这种约束是否可以减少ICS是未知的。因此，本文提出了一种测量ICS通过使用地球移动器（EM）的距离，然后推导出的上限和下限的措施，提供一个理论分析BN。上限表明，BN技术可以控制ICS只有在低尺寸和小噪声的输出，而在其他情况下，他们的控制是无效的。本文还证明了这种控制只是ICS的一个边界，而不是ICS的一个约简。同时，分析表明，高阶矩和噪声，BN无法控制，对下界的影响很大。在此基础上，本文进一步提出了一种算法，该算法将输出与一个可调参数进行单位化，以进一步约束ICS，以应对BN问题。所提出的单位化的上限是无噪声的，只占主导地位的参数。因此，可以训练参数以调整界限并进一步控制ICS。此外，单元化嵌入BN的框架，以减少信息损失。在CIFAR-10、CIFAR-100和ImageNet数据集上的实验表明，该算法的性能优于现有的BN技术。
英文摘要：
Batch Normalization (BN) techniques have been proposed to reduce the so-called Internal Covariate Shift (ICS) by attempting to keep the distributions of layer outputs unchanged. Experiments have shown their effectiveness on training deep neural networks. However, since only the first two moments are controlled in these BN techniques, it seems that a weak constraint is imposed on layer distributions and furthermore whether such constraint can reduce ICS is unknown. Thus this paper proposes a measure for ICS by using the Earth Mover (EM) distance and then derives the upper and lower bounds for the measure to provide a theoretical analysis of BN. The upper bound has shown that BN techniques can control ICS only for the outputs with low dimensions and small noise whereas their control is NOT effective in other cases. This paper also proves that such control is just a bounding of ICS rather than a reduction of ICS. Meanwhile, the analysis shows that the high-order moments and noise, which BN cannot control, have great impact on the lower bound. Based on such analysis, this paper furthermore proposes an algorithm that unitizes the outputs with an adjustable parameter to further bound ICS in order to cope with the problems of BN. The upper bound for the proposed unitization is noise-free and only dominated by the parameter. Thus, the parameter can be trained to tune the bound and further to control ICS. Besides, the unitization is embedded into the framework of BN to reduce the information loss. The experiments show that this proposed algorithm outperforms existing BN techniques on CIFAR-10, CIFAR-100 and ImageNet datasets.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
