meProp:稀疏反向传播,用于加速深度学习,减少过拟合
meProp: Sparsified Back Propagation for Accelerated Deep Learning with Reduced Overfitting
摘要:
我们提出了一个简单而有效的神经网络学习技术。前向传播照常计算。在反向传播中,仅计算完整梯度的一小部分来更新模型参数。梯度向量以这样一种方式稀疏化,即仅保留顶部$k$元素(就幅度而言)。结果,仅修改权重矩阵的$k$行或列(取决于布局),导致计算成本的线性减少($k$除以向量维度)。令人惊讶的是,实验结果表明,我们只能在每个反向传播通道更新1-4%的权重。这不会导致大量的训练迭代。更有趣的是,所得到的模型的准确性实际上得到了提高,而不是降低,并给出了详细的分析。代码可在https://github.com/lancopku/meProp上获得
英文摘要:
We propose a simple yet effective technique for neural network learning. The forward propagation is computed as usual. In back propagation, only a small subset of the full gradient is computed to update the model parameters. The gradient vectors are sparsified in such a way that only the top-$k$ elements (in terms of magnitude) are kept. As a result, only $k$ rows or columns (depending on the layout) of the weight matrix are modified, leading to a linear reduction ($k$ divided by the vector dimension) in the computational cost. Surprisingly, experimental results demonstrate that we can update only 1-4% of the weights at each back propagation pass. This does not result in a larger number of training iterations. More interestingly, the accuracy of the resulting models is actually improved rather than degraded, and a detailed analysis is given. The code is available at https://github.com/lancopku/meProp
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。