克服任务无关持续学习中的增长性遗忘
Overcoming Growth-Induced Forgetting in Task-Agnostic Continual Learning
AI总结：
SparseGrow通过梯度和参数稀疏性解决持续学习中的增长性遗忘问题，提升模型适应性与知识保留能力。
摘要：
在持续学习（CL）中，模型增长增强了对新数据的适应性。然而，当模型增长被不当应用时，尤其是在任务无关的CL中，整个增长后的模型被用于推理时，会导致已学习知识的严重退化，我们称之为增长性遗忘。大多数采用模型增长来提高适应性的现有方法往往忽视遗忘问题，导致知识保留受损，使其不适合任务无关的设置。为了在模型增长中促进适应性和知识保留，我们识别出关键点：梯度和参数稀疏性。引入SparseGrow，通过层扩展和梯度门控增加梯度稀疏性，以实现对参数的聚焦更新，同时保留关键参数，从而抑制遗忘。此外，它通过稀疏初始化和训练促进参数稀疏性，旨在更好地控制模型可塑性，提高对新数据的适应性。在多样化的数据集、任务无关的设置以及大量任务上的广泛实验验证了受控层扩展的必要性，并验证了SparseGrow在实现高适应性的同时最小化遗忘在持续学习中的有效性。通过启用稀疏梯度和参数的模型增长，SparseGrow为构建可扩展的终身学习系统铺平了道路，这些系统能够通过更好的知识保留实现持续适应。
英文摘要：
In continual learning (CL), model growth enhances adaptability to new data. However, when model growth is applied improperly, especially in task-agnostic CL, where the entire grown model is used for inference, it can lead to severe degradation of learned knowledge, a problem we term growth-induced forgetting. Most existing methods that adopt model growth to improve adaptability often overlook the forgetting issue, resulting in compromised knowledge retention, making them unsuitable for task-agnostic settings. To promote both adaptability and knowledge retention with model growth, we identify the key: gradient and parameter sparsity. Introducing SparseGrow, which increases gradient sparsity through layer expansion and gradient gating to enable focused updates on parameters while preserving critical parameters, thus inhibiting forgetting. Moreover, it promotes parameter sparsity with sparse initialization and training, aiming at better control of model plasticity, improving adaptability over new data. Extensive experiments across diverse datasets, task-agnostic settings, and a large number of tasks demonstrate the necessity of controlled layer expansion and validate the effectiveness of SparseGrow in achieving high adaptability while minimizing forgetting in continual learning. By enabling model growth with sparsified gradients and parameters, SparseGrow paves the way for building scalable lifelong learning systems capable of continual adaptation with better knowledge retention.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
