首页/论文/2003.11666

arXiv 2003.116662020-03cs.LGcs.DCstat.ML

Pipelined Backpropagation at Scale: Training Large Models without Batches

Cerebras；Tesla（特斯拉）；Cerebras Systems, Inc（Cerebras Systems 公司）；Google（谷歌）

Atli Kosson, Vitaliy Chiley, Abhinav Venigalla, Joel Hestness, Urs Köster

备注：: Proceedings of the 4th MLSys Conference, 2021

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

英文摘要相关论文

英文摘要：

New hardware can substantially increase the speed and efficiency of deep neural network training. To guide the development of future hardware architectures, it is pertinent to explore the hardware and machine learning properties of alternative training algorithms. In this work we evaluate the use of small batch, fine-grained Pipelined Backpropagation, an asynchronous pipeline parallel training algorithm that has significant hardware advantages. We introduce two methods, Spike Compensation and Linear Weight Prediction, that effectively mitigate the downsides caused by the asynchronicity of Pipelined Backpropagation and outperform existing techniques in our setting. We show that appropriate normalization and small batch sizes can also aid training. With our methods, fine-grained Pipelined Backpropagation using a batch size of one can match the accuracy of SGD for multiple networks trained on CIFAR-10 and ImageNet. Simple scaling rules allow the use of existing hyperparameters for traditional training without additional tuning.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

阅读原文

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

本文内容

英文摘要相关论文

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

继续探索

相关论文

Improving the Backpropagation Algorithm with Consequentialism Weight Updates over Mini-Batches

Amirkabir University of Technology（阿米拉比尔理工大学）；Ferdowsi University of Mashhad（马什哈德菲尔多西大学）

Many attempts took place to improve the adaptive filters that can also be useful to improve backpropagation (BP). Normalized least mean squares (NLMS) is one of the most successful algorithms derived from Least mean squares (LMS). However, its extension to multi-layer neural networks has not happened before. Here, we first show that it is possible to consider a multi-layer neural network as a stack of adaptive filters. Additionally, we introduce more comprehensible interpretations of NLMS than the complicated geometric interpretation in affine projection algorithm (APA) for a single fully-conn…

2021-01-02cs.LGstat.ML

Tensor-Based Backpropagation in Neural Networks with Non-Sequential Input

University of Edinburgh（爱丁堡大学）；AndPlus LLC（AndPlus有限责任公司）

Neural networks have been able to achieve groundbreaking accuracy at tasks conventionally considered only doable by humans. Using stochastic gradient descent, optimization in many dimensions is made possible, albeit at a relatively high computational cost. By splitting training data into batches, networks can be distributed and trained vastly more efficiently and with minimal accuracy loss. We have explored the mathematics behind efficiently implementing tensor-based batch backpropagation algorithms. A common approach to batch training is iterating over batch items individually. Explicitly usi…

2017-07-13cs.LG

Pipelined Training with Stale Weights of Deep Convolutional Neural Networks

University of Toronto（多伦多大学）

The growth in the complexity of Convolutional Neural Networks (CNNs) is increasing interest in partitioning a network across multiple accelerators during training and pipelining the backpropagation computations over the accelerators. Existing approaches avoid or limit the use of stale weights through techniques such as micro-batching or weight stashing. These techniques either underutilize of accelerators or increase memory footprint. We explore the impact of stale weights on the statistical efficiency and performance in a pipelined backpropagation scheme that maximizes accelerator utilization…

2019-12-29cs.DCcs.LG

CATERPILLAR: Coarse Grain Reconfigurable Architecture for Accelerating the Training of Deep Neural Networks

Stanford University（斯坦福大学）；Cerebras Systems

Accelerating the inference of a trained DNN is a well studied subject. In this paper we switch the focus to the training of DNNs. The training phase is compute intensive, demands complicated data communication, and contains multiple levels of data dependencies and parallelism. This paper presents an algorithm/architecture space exploration of efficient accelerators to achieve better network convergence rates and higher energy efficiency for training DNNs. We further demonstrate that an architecture with hierarchical support for collective communication semantics provides flexibility in trainin…

2017-06-08cs.DCcs.LGcs.NE

A Theoretical Framework for Inference Learning

UC, Irvine（加州大学欧文分校）；Oxford University（牛津大学）；Forschungszentrum Jülich（于利希研究中心）

Backpropagation (BP) is the most successful and widely used algorithm in deep learning. However, the computations required by BP are challenging to reconcile with known neurobiology. This difficulty has stimulated interest in more biologically plausible alternatives to BP. One such algorithm is the inference learning algorithm (IL). IL has close connections to neurobiological models of cortical function and has achieved equal performance to BP on supervised learning and auto-associative tasks. In contrast to BP, however, the mathematical foundations of IL are not well-understood. Here, we deve…

2022-06-01cs.LGcs.NE

Efficient and Robust Parallel DNN Training through Model Parallelism on Multi-GPU Platform

The training process of Deep Neural Network (DNN) is compute-intensive, often taking days to weeks to train a DNN model. Therefore, parallel execution of DNN training on GPUs is a widely adopted approach to speed up the process nowadays. Due to the implementation simplicity, data parallelism is currently the most commonly used parallelization method. Nonetheless, data parallelism suffers from excessive inter-GPU communication overhead due to frequent weight synchronization among GPUs. Another approach is pipelined model parallelism, which partitions a DNN model among GPUs, and processes multip…

2019-10-28cs.DCcs.LG

更多相关论文收起更多论文

Sampling-based Gradient Regularization for Capturing Long-Term Dependencies in Recurrent Neural Networks

Institute of MMS of NASU（乌克兰国家科学院建模系统研究所）；Center for Cybernetics（控制论中心）

Vanishing (and exploding) gradients effect is a common problem for recurrent neural networks with nonlinear activation functions which use backpropagation method for calculation of derivatives. Deep feedforward neural networks with many hidden layers also suffer from this effect. In this paper we propose a novel universal technique that makes the norm of the gradient stay in the suitable range. We construct a way to estimate a contribution of each training example to the norm of the long-term components of the target function s gradient. Using this subroutine we can construct mini-batches for …

2017-02-13cs.LGcs.NE

循环数据并行：实现深度神经网络高效并行的方法

Cyclic Data Parallelism for Efficient Parallelism of Deep Neural Networks

Sorbonne Université（索邦大学）；CNRS（法国国家科学研究中心）；Flatiron Institute（弗拉蒂伦研究所）

针对现有数据并行方法中内存峰值和梯度通信瓶颈，提出循环数据并行范式，通过顺序执行微批次并统一延迟，实现恒定内存和平衡通信，减少GPU需求，并在CIFAR-10和ImageNet上验证了有效性。

2024-03-13cs.AIcs.DCcs.LG

Training CNNs faster with Dynamic Input and Kernel Downsampling

University of Toronto（多伦多大学）

We reduce training time in convolutional networks (CNNs) with a method that, for some of the mini-batches: a) scales down the resolution of input images via downsampling, and b) reduces the forward pass operations via pooling on the convolution filters. Training is performed in an interleaved fashion; some batches undergo the regular forward and backpropagation passes with original network parameters, whereas others undergo a forward pass with pooled filters and downsampled inputs. Since pooling is differentiable, the gradients of the pooled filters propagate to the original network parameters…

2019-10-15cs.CVcs.LGstat.ML

The Forward-Forward Algorithm: Some Preliminary Investigations

Google Brain（谷歌大脑）

The aim of this paper is to introduce a new learning procedure for neural networks and to demonstrate that it works well enough on a few small problems to be worth further investigation. The Forward-Forward algorithm replaces the forward and backward passes of backpropagation by two forward passes, one with positive (i.e. real) data and the other with negative data which could be generated by the network itself. Each layer has its own objective function which is simply to have high goodness for positive data and low goodness for negative data. The sum of the squared activities in a layer can b…

2022-12-27cs.LG

RAPID：一种面向小语言模型的高效强化学习算法

RAPID: An Efficient Reinforcement Learning Algorithm for Small Language Models

University of Pennsylvania（宾夕法尼亚大学）

针对小语言模型强化学习微调资源消耗大、耗时长的问题，提出RAPID算法，通过大批次推理、小批次离轨策略梯度更新结合组优势估计与重要性加权校正偏差，在三个基准上以更短运行时间实现相近或更优准确率。

2025-10-03cs.LG

剖析一个小型人工神经网络

Dissecting a Small Artificial Neural Network

The University of Georgia（佐治亚大学）；College of Computing, Georgia Tech（佐治亚理工学院计算学院）

本研究剖析实现XOR门的最简神经网络，结合统计物理微正则熵分析其损失曲面与反向传播动力学，揭示学习的退火相变本质及隐藏神经元对熵垒的消除作用。

2025-01-03cond-mat.dis-nncond-mat.stat-mechcs.LG

Parallel Training of Deep Networks with Local Updates

University of Berkeley（加州大学伯克利分校）；Google Research（谷歌研究院）；Graphcore Research（Graphcore研究院）；Graphcore

Deep learning models trained on large data sets have been widely successful in both vision and language domains. As state-of-the-art deep learning architectures have continued to grow in parameter count so have the compute budgets and times required to train them, increasing the need for compute-efficient methods that parallelize training. Two common approaches to parallelize the training of deep networks have been data and model parallelism. While useful, data and model parallelism suffer from diminishing returns in terms of compute efficiency for large batch sizes. In this paper, we investig…

2021-06-15cs.AIcs.LGcs.NE

面向编码单光子信号处理的端侧学习与可重构硬件实现

Towards On-Device Learning and Reconfigurable Hardware Implementation for Encoded Single-Photon Signal Processing

University of Strathclyde（思克莱德大学）

针对传统基于反向传播的DNNs在单光子信号处理中依赖装置参数、需频繁重新训练且存在延迟与存储开销的问题，提出OSOS-ELM在线训练算法并在FPGA和GPU上实现，提升硬件效率且在多案例中验证了精度。

2025-04-12cs.LGeess.SP