arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2210.01987cs.CVcs.LG
收藏

ImpressLearn:通过组合任务实现持续学习

ImpressLearn: Continual Learning via Combined Task Impressions

Center For Data Science(数据科学中心);New York University(纽约大学)

Dhrupad Bhardwaj, Julia Kempe, Artem Vysogorets, Angela M. Teng, Evaristus C. Ezekwem

摘要:

这项工作提出了一种新方法,可以在多个任务上顺序训练深度神经网络,而不会遭受灾难性遗忘,同时赋予它快速适应看不见的任务的能力。从现有的关于网络掩蔽的工作开始(Wortsman等人,2020),我们表明,在随机初始化的骨干网络上简单地学习少量特定于任务的超级掩码(印象)的线性组合就足以保持先前学习的任务的准确性,并在看不见的任务上实现高准确性。与以前的方法相比,我们不需要为每个新任务生成专用的掩码或上下文,而是利用迁移学习来保持每个任务的参数开销较小。我们的工作说明了线性组合个人印象的力量,其中每一个单独的票价很差,以实现性能媲美一个专用的面具。此外,如果使用足够多的印象,即使是来自相同任务的重复印象(同质掩模),在组合时也可以接近异质组合的性能。我们的方法比现有的方法更有效地扩展,通常需要数量级更少的参数,即使在任务身份丢失的情况下也可以不加修改地运行。此外,在推理时没有给出任务标签的情况下,我们的算法给出了Wortsman等人使用的一次性过程的一个通常有利的替代方案,2020.我们评估我们的方法在一些著名的图像分类数据集和网络架构。

英文摘要:

This work proposes a new method to sequentially train deep neural networks on multiple tasks without suffering catastrophic forgetting, while endowing it with the capability to quickly adapt to unseen tasks. Starting from existing work on network masking (Wortsman et al., 2020), we show that simply learning a linear combination of a small number of task-specific supermasks (impressions) on a randomly initialized backbone network is sufficient to both retain accuracy on previously learned tasks, as well as achieve high accuracy on unseen tasks. In contrast to previous methods, we do not require to generate dedicated masks or contexts for each new task, instead leveraging transfer learning to keep per-task parameter overhead small. Our work illustrates the power of linearly combining individual impressions, each of which fares poorly in isolation, to achieve performance comparable to a dedicated mask. Moreover, even repeated impressions from the same task (homogeneous masks), when combined, can approach the performance of heterogeneous combinations if sufficiently many impressions are used. Our approach scales more efficiently than existing methods, often requiring orders of magnitude fewer parameters and can function without modification even when task identity is missing. In addition, in the setting where task labels are not given at inference, our algorithm gives an often favorable alternative to the one-shot procedure used by Wortsman et al., 2020. We evaluate our method on a number of well-known image classification datasets and network architectures.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑