用于持续学习的独占超掩码子网络训练
Exclusive Supermask Subnetwork Training for Continual Learning
AI总结：
本文提出 ExSSNeT，通过独占且不重叠的子网络权重训练和基于 KNN 的知识迁移模块，在防止灾难性遗忘的同时提升持续学习性能，并在 NLP 和视觉任务上优于 SupSup 等已有方法。
摘要：
持续学习（CL）方法关注在时间推移中积累知识，同时避免灾难性遗忘。最近，Wortsman 等人（2020）提出了一种 CL 方法 SupSup，该方法使用一个随机初始化且固定的基础网络（模型），并为每个新任务寻找一个超掩码，以选择性地保留或移除每个权重，从而产生一个子网络。由于网络权重不被更新，他们能够防止遗忘。尽管不存在遗忘，但 SupSup 的性能并非最优，因为固定权重限制了其表示能力。此外，当学习新任务时，模型内部没有知识积累或迁移。因此，我们提出 ExSSNeT（Exclusive Supermask SubNEtwork Training，独占超掩码子网络训练），该方法执行独占且不重叠的子网络权重训练。这避免了后续任务对共享权重的冲突更新，从而在提升性能的同时仍然防止遗忘。此外，我们提出了一种新颖的基于 KNN 的知识迁移（KKT）模块，该模块利用先前获得的知识来更好、更快地学习新任务。我们证明，ExSSNeT 在 NLP 和视觉领域均优于以往表现强劲的方法，同时防止遗忘。此外，ExSSNeT 对于激活 2-10% 模型参数的稀疏掩码尤其有利，相比 SupSup 平均提升 8.3%。并且，ExSSNeT 可扩展到大量任务（100 个）。我们的代码可在 https://github.com/prateeky2806/exessnet 获取。
英文摘要：
Continual Learning (CL) methods focus on accumulating knowledge over time while avoiding catastrophic forgetting. Recently, Wortsman et al. (2020) proposed a CL method, SupSup, which uses a randomly initialized, fixed base network (model) and finds a supermask for each new task that selectively keeps or removes each weight to produce a subnetwork. They prevent forgetting as the network weights are not being updated. Although there is no forgetting, the performance of SupSup is sub-optimal because fixed weights restrict its representational power. Furthermore, there is no accumulation or transfer of knowledge inside the model when new tasks are learned. Hence, we propose ExSSNeT (Exclusive Supermask SubNEtwork Training), that performs exclusive and non-overlapping subnetwork weight training. This avoids conflicting updates to the shared weights by subsequent tasks to improve performance while still preventing forgetting. Furthermore, we propose a novel KNN-based Knowledge Transfer (KKT) module that utilizes previously acquired knowledge to learn new tasks better and faster. We demonstrate that ExSSNeT outperforms strong previous methods on both NLP and Vision domains while preventing forgetting. Moreover, ExSSNeT is particularly advantageous for sparse masks that activate 2-10% of the model parameters, resulting in an average improvement of 8.3% over SupSup. Furthermore, ExSSNeT scales to a large number of tasks (100). Our code is available at https://github.com/prateeky2806/exessnet.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
