arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2003.12327cs.CVcs.LG
收藏

间歇增白的随机性研究

An Investigation into the Stochasticity of Batch Whitening

Inception Institute of Artificial Intelligence (IIAI)(起源人工智能研究院)

Lei Huang, Lei Zhao, Yi Zhou, Fan Zhu, Li Liu, Ling Shao

摘要:

批量归一化(BN)通过在小批量内执行标准化而被广泛用于各种网络架构中。对这个过程的全面理解一直是深度学习社区的中心目标。不同于现有的作品,通常只分析标准化操作,本文研究了更一般的批白化(BW)。我们的工作源于这样的观察,即虽然各种白化变换等效地改善了条件反射,但它们在区分场景和训练生成对抗网络(GANs)中表现出显着不同的行为。我们把这种现象归因于BW引入的随机性。我们定量地研究了不同白化变换的随机性,并表明它与训练过程中的优化行为有很好的相关性。我们还研究了如何随机性与人口统计推断过程中的估计。基于我们的分析,我们提供了一个框架,设计和比较BW算法在不同的情况下。我们提出的BW算法在ImageNet分类上显著提高了残差网络。此外,我们发现BW的随机性可以提高GAN的性能,但牺牲了训练的稳定性。

英文摘要:

Batch Normalization (BN) is extensively employed in various network architectures by performing standardization within mini-batches. A full understanding of the process has been a central target in the deep learning communities. Unlike existing works, which usually only analyze the standardization operation, this paper investigates the more general Batch Whitening (BW). Our work originates from the observation that while various whitening transformations equivalently improve the conditioning, they show significantly different behaviors in discriminative scenarios and training Generative Adversarial Networks (GANs). We attribute this phenomenon to the stochasticity that BW introduces. We quantitatively investigate the stochasticity of different whitening transformations and show that it correlates well with the optimization behaviors during training. We also investigate how stochasticity relates to the estimation of population statistics during inference. Based on our analysis, we provide a framework for designing and comparing BW algorithms in different scenarios. Our proposed BW algorithm improves the residual networks by a significant margin on ImageNet classification. Besides, we show that the stochasticity of BW can improve the GAN's performance with, however, the sacrifice of the training stability.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑