BackLink:具有反向链接的监督本地培训
BackLink: Supervised Local Training with Backward Links
摘要:
在反向传播(BP)算法的支持下,深度神经网络在解决各种认知任务方面占据了主导地位。标准BP算法中的训练模式受限,需要进行端到端的错误传播,造成大量的内存开销,阻碍了模型的并行化。现有的局部训练方法旨在通过完全切断模块之间的反向路径并隔离其梯度来解决训练障碍,以减少内存开销并加速训练过程。这些方法防止错误在模块之间流动,从而防止信息交换,导致性能低下。本文提出了一种新的局部训练算法BackLink,它引入了模块间的向后依赖,允许错误在模块之间流动。该算法有助于信息随着网络向后流动。为了保持本地训练的计算优势,BackLink限制了模块内的错误传播长度。在各种深度卷积神经网络中进行的大量实验表明,与其他方法相比,我们的方法始终提高了局部训练算法的分类性能。例如,在具有16个局部模块的ResNet 32中,我们的方法在CIFAR 10上的准确率分别超过传统的贪婪局部训练方法4.00%和最近的工作1.83%。计算成本的分析表明,在GPU内存成本和多个GPU上的运行时间方面产生的开销很小。与标准BP算法相比,该算法在ResNet 110上的内存开销减少了79%,仿真运行时间减少了52%.因此,我们的方法可以为提高训练算法的效率和生物相容性创造新的机会。
英文摘要:
Empowered by the backpropagation (BP) algorithm, deep neural networks have dominated the race in solving various cognitive tasks. The restricted training pattern in the standard BP requires end-to-end error propagation, causing large memory cost and prohibiting model parallelization. Existing local training methods aim to resolve the training obstacle by completely cutting off the backward path between modules and isolating their gradients to reduce memory cost and accelerate the training process. These methods prevent errors from flowing between modules and hence information exchange, resulting in inferior performance. This work proposes a novel local training algorithm, BackLink, which introduces inter-module backward dependency and allows errors to flow between modules. The algorithm facilitates information to flow backward along with the network. To preserve the computational advantage of local training, BackLink restricts the error propagation length within the module. Extensive experiments performed in various deep convolutional neural networks demonstrate that our method consistently improves the classification performance of local training algorithms over other methods. For example, in ResNet32 with 16 local modules, our method surpasses the conventional greedy local training method by 4.00\% and a recent work by 1.83\% in accuracy on CIFAR10, respectively. Analysis of computational costs reveals that small overheads are incurred in GPU memory costs and runtime on multiple GPUs. Our method can lead up to a 79\% reduction in memory cost and 52\% in simulation runtime in ResNet110 compared to the standard BP. Therefore, our method could create new opportunities for improving training algorithms towards better efficiency and biological plausibility.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。