迈向持续学习中的无冗余子网络
Towards Redundancy-Free Sub-networks in Continual Learning
AI总结：
针对持续学习中的灾难性遗忘，本文提出信息瓶颈掩码子网络（IBM），通过将重要信息累积到关键权重来构建无冗余子网络，在显著减少参数量和训练时间的同时超越现有参数隔离方法。
摘要：
灾难性遗忘（CF）是持续学习中的一个突出问题。参数隔离通过为每个任务掩码一个子网络来缓解对旧任务的干扰，从而应对这一挑战。然而，这些子网络的构建依赖于权重幅值，而权重幅值并不一定对应权重的重要性，导致保留不重要的权重并构建出冗余的子网络。为了克服这一局限，受信息瓶颈（information bottleneck）的启发——该方法可消除相邻网络层之间的冗余——我们提出信息瓶颈掩码子网络（Information Bottleneck Masked sub-network，IBM），以消除子网络内部的冗余。具体而言，IBM 将有价值的信息累积到关键权重中，以构建无冗余的子网络，不仅通过冻结子网络有效缓解 CF，还通过有价值知识的迁移促进新任务的训练。此外，IBM 对隐藏表示进行分解，使构建过程自动化并具有灵活性。大量实验表明，IBM 始终优于最先进的方法。值得注意的是，IBM 在子网络内参数量减少 70% 且训练时间减少 80% 的情况下，超越了最先进的参数隔离方法。
英文摘要：
Catastrophic Forgetting (CF) is a prominent issue in continual learning. Parameter isolation addresses this challenge by masking a sub-network for each task to mitigate interference with old tasks. However, these sub-networks are constructed relying on weight magnitude, which does not necessarily correspond to the importance of weights, resulting in maintaining unimportant weights and constructing redundant sub-networks. To overcome this limitation, inspired by information bottleneck, which removes redundancy between adjacent network layers, we propose \textbf{\underline{I}nformation \underline{B}ottleneck \underline{M}asked sub-network (IBM)} to eliminate redundancy within sub-networks. Specifically, IBM accumulates valuable information into essential weights to construct redundancy-free sub-networks, not only effectively mitigating CF by freezing the sub-networks but also facilitating new tasks training through the transfer of valuable knowledge. Additionally, IBM decomposes hidden representations to automate the construction process and make it flexible. Extensive experiments demonstrate that IBM consistently outperforms state-of-the-art methods. Notably, IBM surpasses the state-of-the-art parameter isolation method with a 70\% reduction in the number of parameters within sub-networks and an 80\% decrease in training time.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
