基于动态稀疏训练的持续学习：探索有效模型更新算法
Continual Learning with Dynamic Sparse Training: Exploring Algorithms for Effective Model Updates
AI总结：
本文首次实证研究动态稀疏训练各组件在持续学习中的作用，在CIFAR100和miniImageNet上分析初始化、增长策略与自适应性，发现不同稀疏度下最优配置不同，自适应性有助于提升持续学习性能。
摘要：
持续学习（CL）指智能系统以尽可能少的计算开销，从数据流中顺序获取并保留知识的能力。为此，文献中引入了正则化、重放、架构和参数隔离等方法。参数隔离使用稀疏网络，能够将神经网络的不同部分分配给不同任务，并且如果任务相似，还允许任务之间共享参数。动态稀疏训练（DST）是寻找这些稀疏网络并为每个任务隔离它们的一种重要方法。本文是首个实证研究，旨在考察CL范式下不同DST组件的影响，以填补关键研究空白，并揭示DST用于CL的最优配置（如果存在）。因此，我们进行了一项综合研究，在任务增量式CL设置下，于著名的CIFAR100和miniImageNet基准上研究各种DST组件，以找到每个任务的最佳拓扑，因为我们的主要重点是评估各种DST准则的性能，而非掩码选择过程。我们发现，在低稀疏度水平下，Erdos-Rényi Kernel（ERK）初始化能更高效地利用骨干网络，并允许有效学习任务增量。在高稀疏度水平下，除非是极端稀疏，均匀初始化表现出更可靠和稳健的性能。在增长策略方面，性能取决于所定义的初始化策略和稀疏程度。最后，DST组件内的自适应性是获得更好持续学习者的一个有前景的方向。
英文摘要：
Continual learning (CL) refers to the ability of an intelligent system to sequentially acquire and retain knowledge from a stream of data with as little computational overhead as possible. To this end; regularization, replay, architecture, and parameter isolation approaches were introduced to the literature. Parameter isolation using a sparse network which enables to allocate distinct parts of the neural network to different tasks and also allows to share of parameters between tasks if they are similar. Dynamic Sparse Training (DST) is a prominent way to find these sparse networks and isolate them for each task. This paper is the first empirical study investigating the effect of different DST components under the CL paradigm to fill a critical research gap and shed light on the optimal configuration of DST for CL if it exists. Therefore, we perform a comprehensive study in which we investigate various DST components to find the best topology per task on well-known CIFAR100 and miniImageNet benchmarks in a task-incremental CL setup since our primary focus is to evaluate the performance of various DST criteria, rather than the process of mask selection. We found that, at a low sparsity level, Erdos-Rényi Kernel (ERK) initialization utilizes the backbone more efficiently and allows to effectively learn increments of tasks. At a high sparsity level, unless it is extreme, uniform initialization demonstrates a more reliable and robust performance. In terms of growth strategy; performance is dependent on the defined initialization strategy and the extent of sparsity. Finally, adaptivity within DST components is a promising way for better continual learners.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
