MLAAN：基于多层跳跃增强辅助网络的监督式局部学习扩展
MLAAN: Scaling Supervised Local Learning with Multilaminar Leap Augmented Auxiliary Network
AI总结：
针对DNN端到端训练内存消耗高及局部学习缺乏全局信息交互的问题，提出多层跳跃增强辅助网络MLAAN，通过MLM与LAM协同捕获全局特征并促进模块间信息交换，显著提升性能并降低GPU内存消耗。
摘要：
深度神经网络（DNNs）通常采用端到端（E2E）训练范式，这带来了一些挑战，包括训练期间GPU内存消耗高、效率低下以及模型并行化困难。近期研究试图解决这些问题，其中一种有前景的方法是局部学习。该方法将骨干网络划分为梯度隔离的模块，并手动设计辅助网络来训练这些局部模块。现有方法往往忽略了局部模块之间的信息交互，导致短视问题，并且与E2E训练相比存在性能差距。为了解决这些局限性，我们提出了多层跳跃增强辅助网络（MLAAN）。具体而言，MLAAN由多层局部模块（MLM）和跳跃增强模块（LAM）组成。MLM通过独立和级联的辅助网络捕获局部和全局特征，缓解了由全局特征不足引起的性能问题。然而，过于简化的辅助网络会阻碍MLM捕获全局信息的能力。为此，我们进一步设计了LAM，这是一种增强型辅助网络，使用指数移动平均（EMA）方法促进局部模块之间的信息交换，从而缓解因交互不足导致的短视问题。MLM和LAM之间的协同作用表现出优异的性能。我们在CIFAR-10、STL-10、SVHN和ImageNet数据集上的实验表明，MLAAN可以无缝集成到现有的局部学习框架中，显著提升其性能，甚至超越端到端（E2E）训练方法，同时减少GPU内存消耗。
英文摘要：
Deep neural networks (DNNs) typically employ an end-to-end (E2E) training paradigm which presents several challenges, including high GPU memory consumption, inefficiency, and difficulties in model parallelization during training. Recent research has sought to address these issues, with one promising approach being local learning. This method involves partitioning the backbone network into gradient-isolated modules and manually designing auxiliary networks to train these local modules. Existing methods often neglect the interaction of information between local modules, leading to myopic issues and a performance gap compared to E2E training. To address these limitations, we propose the Multilaminar Leap Augmented Auxiliary Network (MLAAN). Specifically, MLAAN comprises Multilaminar Local Modules (MLM) and Leap Augmented Modules (LAM). MLM captures both local and global features through independent and cascaded auxiliary networks, alleviating performance issues caused by insufficient global features. However, overly simplistic auxiliary networks can impede MLM's ability to capture global information. To address this, we further design LAM, an enhanced auxiliary network that uses the Exponential Moving Average (EMA) method to facilitate information exchange between local modules, thereby mitigating the shortsightedness resulting from inadequate interaction. The synergy between MLM and LAM has demonstrated excellent performance. Our experiments on the CIFAR-10, STL-10, SVHN, and ImageNet datasets show that MLAAN can be seamlessly integrated into existing local learning frameworks, significantly enhancing their performance and even surpassing end-to-end (E2E) training methods, while also reducing GPU memory consumption.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
