稀疏正交参数调优用于持续学习
Sparse Orthogonal Parameters Tuning for Continual Learning
AI总结：
本文提出了一种名为SoTU的新型方法，通过稀疏正交参数调优来解决持续学习中的灾难性遗忘问题，实现了对流数据的最优特征表示。
摘要：
基于预训练模型（PTM）的持续学习方法近年来引起了广泛关注，这些方法能够适应连续的下游任务而无需灾难性遗忘。这些方法通常不更新预训练参数，而是使用额外的适配器、提示和分类器。在本文中，我们从新的角度研究了稀疏正交参数对持续学习的益处。我们发现，合并来自多个流任务的模型所学习的稀疏正交性在解决灾难性遗忘方面具有巨大潜力。利用这一见解，我们提出了一种新颖且有效的称为SoTU（稀疏正交参数调优）的方法。我们假设SoTU的有效性在于将多个领域学到的知识转换为正交delta参数的融合。在多样化的CL基准测试中评估了所提出的方法的有效性。值得注意的是，SoTU在不需要复杂分类器设计的情况下实现了流数据的最优特征表示，使其成为一种即插即用的解决方案。
英文摘要：
Continual learning methods based on pre-trained models (PTM) have recently gained attention which adapt to successive downstream tasks without catastrophic forgetting. These methods typically refrain from updating the pre-trained parameters and instead employ additional adapters, prompts, and classifiers. In this paper, we from a novel perspective investigate the benefit of sparse orthogonal parameters for continual learning. We found that merging sparse orthogonality of models learned from multiple streaming tasks has great potential in addressing catastrophic forgetting. Leveraging this insight, we propose a novel yet effective method called SoTU (Sparse Orthogonal Parameters TUning). We hypothesize that the effectiveness of SoTU lies in the transformation of knowledge learned from multiple domains into the fusion of orthogonal delta parameters. Experimental evaluations on diverse CL benchmarks demonstrate the effectiveness of the proposed approach. Notably, SoTU achieves optimal feature representation for streaming data without necessitating complex classifier designs, making it a Plug-and-Play solution.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
