arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2303.02733cs.LGcs.AIcs.CV
收藏

通过空间梯度缩放的重新参数化

Reparameterization through Spatial Gradient Scaling

University of Alberta(阿尔伯塔大学);Huawei Technologies(华为技术有限公司);Huawei Kirin Solutions(华为麒麟解决方案)

Alexander Detkov, Mohammad Salameh, Muhammad Fetrat Qharabagh, Jialin Zhang, Wei Lui, Shangling Jui, Di Niu

摘要:

重新参数化旨在通过在训练期间将卷积层转换为等效的多分支结构来提高深度神经网络的泛化能力。然而,在理解重新参数化如何改变和有益于神经网络的学习过程方面存在差距。在本文中,我们提出了一种新的空间梯度缩放方法来重新分配卷积网络中的权重之间的学习焦点。我们证明,空间梯度缩放实现了相同的学习动力学分支重新参数化,但没有引入结构变化的网络。我们进一步提出了一种分析方法,该方法基于由互信息测量的输入特征图的空间特征来动态学习每个卷积层的缩放。在CIFAR-10,CIFAR-100和ImageNet上的实验表明,在不搜索重新参数化结构的情况下,我们提出的缩放方法以较低的计算成本优于最先进的重新参数化策略。

英文摘要:

Reparameterization aims to improve the generalization of deep neural networks by transforming convolutional layers into equivalent multi-branched structures during training. However, there exists a gap in understanding how reparameterization may change and benefit the learning process of neural networks. In this paper, we present a novel spatial gradient scaling method to redistribute learning focus among weights in convolutional networks. We prove that spatial gradient scaling achieves the same learning dynamics as a branched reparameterization yet without introducing structural changes into the network. We further propose an analytical approach that dynamically learns scalings for each convolutional layer based on the spatial characteristics of its input feature map gauged by mutual information. Experiments on CIFAR-10, CIFAR-100, and ImageNet show that without searching for reparameterized structures, our proposed scaling method outperforms the state-of-the-art reparameterization strategies at a lower computational cost.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑