arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2010.01546cs.LGstat.ML
收藏

基于梯度变换的特征白化算法

Feature Whitening via Gradient Transformation for Improved Convergence

Intel(英特尔)

Shmulik Markovich-Golan, Barak Battash, Amit Bleiweiss

摘要:

特征白化是用于加速DNN训练的已知技术。在某些假设下,白化激活将Fisher信息矩阵减少为简单的单位矩阵,在这种情况下,随机梯度下降等同于更快的自然梯度下降。由于在前向和后向传播中变换层输入及其对应的梯度以及重复计算特征值分解(EVD)导致的额外复杂性,该方法迄今为止并不常用。在这项工作中,我们解决的复杂性缺陷的特征白化。我们的贡献是双重的。首先,我们推导出一个等价的方法,该方法将样本变换替换为对权重梯度的变换,并应用于每批B样本。复杂度降低了因子S=(2B),其中S表示层输出的特征维度。随着批量大小随着分布式训练的增加而增加,使用所提出的方法的好处变得更加引人注目。其次,根据样本协方差矩阵的条件数与收敛速度之间的理论关系,推导出一种递归地减少后一矩阵的条件数的次优算法。与EVD相比,复杂度降低了输入特征维度M的一个因子。我们将提出的算法与基于ResNet的图像分类网络在CIFAR和Imagenet数据集上进行了验证。所提出的算法是简单的,我们实现了一个分布式版本。在我们的实验中,可以观察到在速度和达到的精度方面的改进的收敛。

英文摘要:

Feature whitening is a known technique for speeding up training of DNN. Under certain assumptions, whitening the activations reduces the Fisher information matrix to a simple identity matrix, in which case stochastic gradient descent is equivalent to the faster natural gradient descent. Due to the additional complexity resulting from transforming the layer inputs and their corresponding gradients in the forward and backward propagation, and from repeatedly computing the Eigenvalue decomposition (EVD), this method is not commonly used to date. In this work, we address the complexity drawbacks of feature whitening. Our contribution is twofold. First, we derive an equivalent method, which replaces the sample transformations by a transformation to the weight gradients, applied to every batch of B samples. The complexity is reduced by a factor of S=(2B), where S denotes the feature dimension of the layer output. As the batch size increases with distributed training, the benefit of using the proposed method becomes more compelling. Second, motivated by the theoretical relation between the condition number of the sample covariance matrix and the convergence speed, we derive an alternative sub-optimal algorithm which recursively reduces the condition number of the latter matrix. Compared to EVD, complexity is reduced by a factor of the input feature dimension M. We exemplify the proposed algorithms with ResNet-based networks for image classification demonstrated on the CIFAR and Imagenet datasets. Parallelizing the proposed algorithms is straightforward and we implement a distributed version thereof. Improved convergence, in terms of speed and attained accuracy, can be observed in our experiments.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑