arXiv 1707.045852017-07-14cs.CVcs.LG

The Reversible Residual Network: Backpropagation Without Storing Activations

University of Toronto（多伦多大学）；Uber Advanced Technologies Group（Uber先进技术集团）

Aidan N. Gomez, Mengye Ren, Raquel Urtasun, Roger B. Grosse

英文摘要：

Deep residual networks (ResNets) have significantly pushed forward the state-of-the-art on image classification, increasing in performance as networks grow both deeper and wider. However, memory consumption becomes a bottleneck, as one needs to store the activations in order to calculate gradients using backpropagation. We present the Reversible Residual Network (RevNet), a variant of ResNets where each layer's activations can be reconstructed exactly from the next layer's. Therefore, the activations for most layers need not be stored in memory during backpropagation. We demonstrate the effectiveness of RevNets on CIFAR-10, CIFAR-100, and ImageNet, establishing nearly identical classification accuracy to equally-sized ResNets, even though the activation storage requirements are independent of depth.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

阅读原文

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

本文内容

英文摘要相关论文

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

继续探索

相关论文

PaReprop：快速并行化可逆反向传播

PaReprop: Fast Parallelized Reversible Backpropagation

UC Berkeley（加州大学伯克利分校）

针对可逆Transformer训练中反向传播阶段激活重计算带来的额外开销问题，提出PaReprop并行化可逆反向传播算法，将激活重计算与梯度计算并行，经多模态多模型基准测试，训练吞吐量最高比普通可逆训练提升20%。

2023-06-15cs.CLcs.CVcs.LG

ParaRevSNN：一种用于高效训练和推理的并行可逆脉冲神经网络

ParaRevSNN: A Parallel Reversible Spiking Neural Network for Efficient Training and Inference

针对可逆脉冲神经网络顺序计算导致的高延迟问题，提出并行可逆架构ParaRevSNN，通过解耦块间依赖实现并行，在保持精度的同时显著降低训练和推理时间。

2025-08-02cs.CV

A Partially Reversible U-Net for Memory-Efficient Volumetric Image Segmentation

ETH Zürich（苏黎世联邦理工学院）

One of the key drawbacks of 3D convolutional neural networks for segmentation is their memory footprint, which necessitates compromises in the network architecture in order to fit into a given memory budget. Motivated by the RevNet for image classification, we propose a partially reversible U-Net architecture that reduces memory consumption substantially. The reversible architecture allows us to exactly recover each layer's outputs from the subsequent layer's ones, eliminating the need to store activations for backpropagation. This alleviates the biggest memory bottleneck and enables very deep…

2019-06-20cs.CVeess.IV

Fully reversible neural networks for large-scale surface and sub-surface characterization via remote sensing

Computational Geosciences Inc.（计算地球科学公司）；University of British Columbia（不列颠哥伦比亚大学）

The large spatial/frequency scale of hyperspectral and airborne magnetic and gravitational data causes memory issues when using convolutional neural networks for (sub-) surface characterization. Recently developed fully reversible networks can mostly avoid memory limitations by virtue of having a low and fixed memory requirement for storing network states, as opposed to the typical linear memory growth with depth. Fully reversible networks enable the training of deep neural networks that take in entire data volumes, and create semantic segmentations in one go. This approach avoids the need to …

2020-03-16cs.CVeess.IVphysics.geo-ph

Mesa: A Memory-saving Training Framework for Transformers

Monash University（莫纳什大学）

There has been an explosion of interest in designing high-performance Transformers. While Transformers have delivered significant performance improvements, training such networks is extremely memory intensive owing to storing all intermediate activations that are needed for gradient computation during backpropagation, especially for long sequences. To this end, we present Mesa, a memory-saving training framework for Transformers. Specifically, Mesa uses exact activations during forward pass while storing a low-precision version of activations to reduce memory consumption during training. The l…

2022-08-29cs.CVcs.LG

Reversible designs for extreme memory cost reduction of CNN training

Sicara（西卡拉公司）

Training Convolutional Neural Networks (CNN) is a resource intensive task that requires specialized hardware for efficient computation. One of the most limiting bottleneck of CNN training is the memory cost associated with storing the activation values of hidden layers needed for the computation of the weights gradient during the backward pass of the backpropagation algorithm. Recently, reversible architectures have been proposed to reduce the memory cost of training large CNN by reconstructing the input activation values of hidden layers from their output during the backward pass, circumventi…

2019-10-24cs.CV

更多相关论文收起更多论文

Momentum Capsule Networks

University of Innsbruck（因斯布鲁克大学）；DeepOpinion（DeepOpinion公司）

Capsule networks are a class of neural networks that achieved promising results on many computer vision tasks. However, baseline capsule networks have failed to reach state-of-the-art results on more complex datasets due to the high computation and memory requirements. We tackle this problem by proposing a new network architecture, called Momentum Capsule Network (MoCapsNet). MoCapsNets are inspired by Momentum ResNets, a type of network that applies reversible residual building blocks. Reversible networks allow for recalculating activations of the forward pass in the backpropagation algorithm…

2022-08-25cs.AIcs.CVcs.LG

RevPHiSeg: A Memory-Efficient Neural Network for Uncertainty Quantification in Medical Image Segmentation

Computer Vision Laboratory, ETH Zürich（苏黎世联邦理工学院计算机视觉实验室）

Quantifying segmentation uncertainty has become an important issue in medical image analysis due to the inherent ambiguity of anatomical structures and its pathologies. Recently, neural network-based uncertainty quantification methods have been successfully applied to various problems. One of the main limitations of the existing techniques is the high memory requirement during training; which limits their application to processing smaller field-of-views (FOVs) and/or using shallower architectures. In this paper, we investigate the effect of using reversible blocks for building memory-efficient…

2020-08-18cs.CVeess.IV

Feed-Forward On-Edge Fine-tuning Using Static Synthetic Gradient Modules

KU Leuven（鲁汶大学）

Training deep learning models on embedded devices is typically avoided since this requires more memory, computation and power over inference. In this work, we focus on lowering the amount of memory needed for storing all activations, which are required during the backward pass to compute the gradients. Instead, during the forward pass, static Synthetic Gradient Modules (SGMs) predict gradients for each layer. This allows training the model in a feed-forward manner without having to store all activations. We tested our method on a robot grasping scenario where a robot needs to learn to grasp ne…

2020-09-21cs.CV

Training DNNs in O(1) memory with MEM-DFA using Random Matrices

Warsaw University（华沙大学）；Tensorflight, Inc.（Tensorflight公司）

This work presents a method for reducing memory consumption to a constant complexity when training deep neural networks. The algorithm is based on the more biologically plausible alternatives of the backpropagation (BP): direct feedback alignment (DFA) and feedback alignment (FA), which use random matrices to propagate error. The proposed method, memory-efficient direct feedback alignment (MEM-DFA), uses higher independence of layers in DFA and allows avoiding storing at once all activation vectors, unlike standard BP, FA, and DFA. Thus, our algorithm's memory usage is constant regardless of t…

2020-12-21cs.CV

Invertible Residual Network with Regularization for Effective Medical Image Segmentation

University of Arkansas（阿肯色大学）

Deep Convolutional Neural Networks (CNNs) i.e. Residual Networks (ResNets) have been used successfully for many computer vision tasks, but are difficult to scale to 3D volumetric medical data. Memory is increasingly often the bottleneck when training 3D Convolutional Neural Networks (CNNs). Recently, invertible neural networks have been applied to significantly reduce activation memory footprint when training neural networks with backpropagation thanks to the invertible functions that allow retrieving its input from its output without storing intermediate activations in memory to perform the b…

2021-03-16cs.CVeess.IV

GLEAM: Greedy Learning for Large-Scale Accelerated MRI Reconstruction

Stanford University（斯坦福大学）；NVIDIA Inc（英伟达公司）

Unrolled neural networks have recently achieved state-of-the-art accelerated MRI reconstruction. These networks unroll iterative optimization algorithms by alternating between physics-based consistency and neural-network based regularization. However, they require several iterations of a large neural network to handle high-dimensional imaging tasks such as 3D MRI. This limits traditional training algorithms based on backpropagation due to prohibitively large memory and compute requirements for calculating gradients and storing intermediate activations. To address this challenge, we propose Gre…

2022-07-18cs.CVeess.IV

用于大规模地表与地下数据分割的完全可逆双曲神经网络

Fully invertible hyperbolic neural networks for segmenting large-scale surface and sub-surface data

Computational Geosciences Inc.（计算地球科学公司）；The University of British Columbia（不列颠哥伦比亚大学）

本文提出基于电报方程的完全可逆双曲神经网络，通过压缩卷积核和以非标准方式使用可逆网络，解决大规模地表与地下数据分割中的内存限制和输入输出维度不匹配问题，并在高光谱分类、航空物探和地震成像中验证了可一次性处理大数据量的能力。

2024-06-30cs.CVphysics.geo-ph