LLS:受神经活动同步现象启发的深度神经网络局部学习规则
LLS: Local Learning Rule for Deep Neural Networks Inspired by Neural Activity Synchronization
AI总结:
本文提出受大脑神经活动同步现象启发的局部学习规则LLS,利用固定周期基向量同步层内神经元活动,在图像分类任务上以最多减少300倍MAC运算和一半内存需求达到与反向传播相当的性能,适用于设备端学习。
摘要:
使用传统反向传播(BP)训练深度神经网络(DNN)在计算复杂度和能耗方面面临挑战,尤其是在计算资源有限的设备端学习场景中。为了应对这些挑战,人们已经探索了多种BP的替代方案,包括随机反馈对齐、前向-前向算法和局部分类器。这些方法各有优势,但在处理复杂的视觉任务时可能会遇到困难,或者需要消耗大量计算资源。在本文中,我们提出了一种新颖的局部学习规则,其灵感来源于大脑中观察到的神经活动同步现象(LLS)。LLS利用固定的周期基向量来同步每一层内的神经元活动,从而无需额外的可训练参数即可实现高效训练。我们在多个图像分类数据集上展示了LLS及其变体LLS-M和LLS-MxM的有效性,在降低计算复杂度和仅增加极少额外参数的情况下,达到了与BP相当的准确率。具体而言,LLS在达到相当性能的同时,乘加(MAC)运算次数最多可减少300倍,内存需求仅为BP的一半。此外,LLS在Visual Wake Word(VWW)数据集上的表现凸显了其适用于设备端学习任务的能力,使其成为边缘硬件实现的一个有前景的候选方案。
英文摘要:
Training deep neural networks (DNNs) using traditional backpropagation (BP) presents challenges in terms of computational complexity and energy consumption, particularly for on-device learning where computational resources are limited. Various alternatives to BP, including random feedback alignment, forward-forward, and local classifiers, have been explored to address these challenges. These methods have their advantages, but they can encounter difficulties when dealing with intricate visual tasks or demand considerable computational resources. In this paper, we propose a novel Local Learning rule inspired by neural activity Synchronization phenomena (LLS) observed in the brain. LLS utilizes fixed periodic basis vectors to synchronize neuron activity within each layer, enabling efficient training without the need for additional trainable parameters. We demonstrate the effectiveness of LLS and its variations, LLS-M and LLS-MxM, on multiple image classification datasets, achieving accuracy comparable to BP with reduced computational complexity and minimal additional parameters. Specifically, LLS achieves comparable performance with up to $300 \times$ fewer multiply-accumulate (MAC) operations and half the memory requirements of BP. Furthermore, the performance of LLS on the Visual Wake Word (VWW) dataset highlights its suitability for on-device learning tasks, making it a promising candidate for edge hardware implementations.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。