无偏近似向量-雅可比积以提高反向传播效率
Unbiased Approximate Vector-Jacobian Products for Efficient Backpropagation
AI总结：
本文提出了一种无偏随机反向传播方法，通过近似向量-雅可比积来降低深度学习的计算和内存成本，并通过实验验证了其有效性。
摘要：
在本工作中，我们介绍了减少训练深度神经网络计算和内存成本的方法。我们的方法是在反向传播过程中用随机、无偏的近似值替代精确的向量-雅可比积。我们对达到目标精度所需的epoch数量与每个epoch的成本减少之间的权衡进行了理论分析。然后，我们识别出特定的无偏向量-雅可比积估计，并建立了在稀疏约束下具有最优性质的最小方差属性。最后，我们在多层感知机、BagNets和视觉Transformer架构上进行了深入实验。这些实验验证了我们的理论结果，并确认了我们提出的无偏随机反向传播方法在降低深度学习成本方面的潜力。
英文摘要：
In this work we introduce methods to reduce the computational and memory costs of training deep neural networks. Our approach consists in replacing exact vector-jacobian products by randomized, unbiased approximations thereof during backpropagation. We provide a theoretical analysis of the trade-off between the number of epochs needed to achieve a target precision and the cost reduction for each epoch. We then identify specific unbiased estimates of vector-jacobian products for which we establish desirable optimality properties of minimal variance under sparsity constraints. Finally we provide in-depth experiments on multi-layer perceptrons, BagNets and Visual Transfomers architectures. These validate our theoretical results, and confirm the potential of our proposed unbiased randomized backpropagation approach for reducing the cost of deep learning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
