arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1706.06197cs.LGcs.AIcs.CLcs.CV
收藏

meProp:稀疏反向传播,用于加速深度学习,减少过拟合

meProp: Sparsified Back Propagation for Accelerated Deep Learning with Reduced Overfitting

School of Electronics Engineering and Computer Science, Peking University(北京大学电子工程与计算机科学学院);MOE Key Laboratory of Computational Linguistics, Peking University(北京大学计算语言学教育部重点实验室)

Xu Sun, Xuancheng Ren, Shuming Ma, Houfeng Wang

摘要:

我们提出了一个简单而有效的神经网络学习技术。前向传播照常计算。在反向传播中,仅计算完整梯度的一小部分来更新模型参数。梯度向量以这样一种方式稀疏化,即仅保留顶部$k$元素(就幅度而言)。结果,仅修改权重矩阵的$k$行或列(取决于布局),导致计算成本的线性减少($k$除以向量维度)。令人惊讶的是,实验结果表明,我们只能在每个反向传播通道更新1-4%的权重。这不会导致大量的训练迭代。更有趣的是,所得到的模型的准确性实际上得到了提高,而不是降低,并给出了详细的分析。代码可在https://github.com/lancopku/meProp上获得

英文摘要:

We propose a simple yet effective technique for neural network learning. The forward propagation is computed as usual. In back propagation, only a small subset of the full gradient is computed to update the model parameters. The gradient vectors are sparsified in such a way that only the top-$k$ elements (in terms of magnitude) are kept. As a result, only $k$ rows or columns (depending on the layout) of the weight matrix are modified, leading to a linear reduction ($k$ divided by the vector dimension) in the computational cost. Surprisingly, experimental results demonstrate that we can update only 1-4% of the weights at each back propagation pass. This does not result in a larger number of training iterations. More interestingly, the accuracy of the resulting models is actually improved rather than degraded, and a detailed analysis is given. The code is available at https://github.com/lancopku/meProp

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑