arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.02052cs.LGstat.ML
收藏

PETRA:基于可逆架构的并行端到端训练

PETRA: Parallel End-to-end Training with Reversible Architectures

Flatiron Institute(熨斗研究所);Mila(米拉研究所)

Stéphane Rivaud, Louis Fournier, Thomas Pumir, Eugene Belilovsky, Michael Eickenberg, Edouard Oyallon

AI总结:

本研究提出基于可逆架构的并行训练方法PETRA,通过解耦前后向传播、消除权重暂存实现高效模型并行,在多个图像数据集上取得与反向传播相当的精度。

摘要:

可逆架构已被证明能够达到与非可逆架构相当的性能,可应用于深度学习以节省内存,还可用于生成建模。在本研究中,我们展示了可逆架构如何解决深度模型训练并行化过程中的挑战。我们提出了PETRA,这是一种用于并行化梯度计算的新型反向传播替代方案。PETRA通过让各阶段(即一组层)在不同设备上独立计算,同时仅需在设备间传递激活值和梯度,实现了高效的模型并行。通过解耦前向传播与反向传播,并仅保留单一版本的更新参数,该方法还消除了权重暂存的需求。我们为PETRA开发了一个类似自动微分的定制训练框架,并在CIFAR-10、ImageNet32和ImageNet数据集上验证了其有效性,使用ResNet-18、ResNet-34和ResNet-50模型取得了与反向传播相当的竞争力精度。

英文摘要:

Reversible architectures have been shown to be capable of performing on par with their non-reversible architectures, being applied in deep learning for memory savings and generative modeling. In this work, we show how reversible architectures can solve challenges in parallelizing deep model training. We introduce PETRA, a novel alternative to backpropagation for parallelizing gradient computations. PETRA facilitates effective model parallelism by enabling stages (i.e., a set of layers) to compute independently on different devices, while only needing to communicate activations and gradients between each other. By decoupling the forward and backward passes and keeping a single updated version of the parameters, the need for weight stashing is also removed. We develop a custom autograd-like training framework for PETRA, and we demonstrate its effectiveness on CIFAR-10, ImageNet32, and ImageNet, achieving competitive accuracies comparable to backpropagation using ResNet-18, ResNet-34, and ResNet-50 models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑