通过子网络间随机局部竞争实现边缘端持续深度学习
Continual Deep Learning on the Edge via Stochastic Local Competition among Subnetworks
AI总结：
针对边缘设备资源受限下的持续学习难题，提出基于子网络单元随机局部竞争的稀疏化方法，同时稀疏化权重与梯度，生成任务特定稀疏表征，为边缘端持续学习提供高效可扩展方案。
摘要：
边缘设备上的持续学习因严格的资源约束带来了独特挑战。本文提出一种利用随机竞争原理促进稀疏性的新方法，可显著降低深度网络的内存占用与计算需求。具体而言，我们提出的深度网络由多个单元块构成，各单元块内的单元会以随机方式开展局部竞争，以赢得对每个新出现任务的表征权。这种网络结构会在每个网络层生成稀疏的任务特定表征；稀疏模式在训练过程中得到，且不同任务的稀疏模式存在差异。关键在于，我们的方法会同时稀疏化权重与权重梯度，从而为边缘设备上的训练提供便利，这一过程基于块内每个单元的获胜概率执行。在推理阶段，网络仅保留获胜单元，并将当前任务对应的非获胜单元的所有权重置零。因此，本方法专门适配边缘设备部署，为资源受限环境下的持续学习提供了高效且可扩展的解决方案。
英文摘要：
Continual learning on edge devices poses unique challenges due to stringent resource constraints. This paper introduces a novel method that leverages stochastic competition principles to promote sparsity, significantly reducing deep network memory footprint and computational demand. Specifically, we propose deep networks that comprise blocks of units that compete locally to win the representation of each arising new task; competition takes place in a stochastic manner. This type of network organization results in sparse task-specific representations from each network layer; the sparsity pattern is obtained during training and is different among tasks. Crucially, our method sparsifies both the weights and the weight gradients, thus facilitating training on edge devices. This is performed on the grounds of winning probability for each unit in a block. During inference, the network retains only the winning unit and zeroes-out all weights pertaining to non-winning units for the task at hand. Thus, our approach is specifically tailored for deployment on edge devices, providing an efficient and scalable solution for continual learning in resource-limited environments.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
