arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1901.07988cs.LGstat.ML
收藏

记忆高效网络训练的近似激活反向传播算法

Backprop with Approximate Activations for Memory-efficient Network Training

Washington University in St. Louis(圣路易斯华盛顿大学);Carnegie Mellon University(卡内基梅隆大学)

Ayan Chakrabarti, Benjamin Moseley

摘要:

训练卷积神经网络模型是内存密集型的,因为反向传播需要存储所有中间层的激活。这在寻求在生产中部署非常深入的架构时提出了一个实际问题,特别是当模型需要在更新的数据集上频繁重新训练时。在本文中,我们提出了一种新的反向传播实现,通过使用计算成本可以忽略不计且对训练性能影响最小的近似值,可以显着减少内存使用。该算法重用公共缓冲区来临时存储完全激活并精确计算前向传递。它还存储激活的近似每层副本,显著节省内存,用于向后传递。与简单地近似标准反向传播中的激活相比,我们的方法限制了跨层误差的积累。这允许使用精度低得多的近似值,而不会影响训练精度。在CIFAR-10、CIFAR-100和ImageNet上的实验表明,我们的方法产生的性能接近精确训练,同时以低至4位的精度存储激活。

英文摘要:

Training convolutional neural network models is memory intensive since back-propagation requires storing activations of all intermediate layers. This presents a practical concern when seeking to deploy very deep architectures in production, especially when models need to be frequently re-trained on updated datasets. In this paper, we propose a new implementation for back-propagation that significantly reduces memory usage, by enabling the use of approximations with negligible computational cost and minimal effect on training performance. The algorithm reuses common buffers to temporarily store full activations and compute the forward pass exactly. It also stores approximate per-layer copies of activations, at significant memory savings, that are used in the backward pass. Compared to simply approximating activations within standard back-propagation, our method limits accumulation of errors across layers. This allows the use of much lower-precision approximations without affecting training accuracy. Experiments on CIFAR-10, CIFAR-100, and ImageNet show that our method yields performance close to exact training, while storing activations compactly with as low as 4-bit precision.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑