KSM:基于核软掩模学习的快速多任务自适应
KSM: Fast Multiple Task Adaption via Kernel-wise Soft Mask Learning
摘要:
深度神经网络(DNN)在学习新任务时可能会忘记先前任务的知识,这被称为\textit{catastrophic forgetting}。虽然最近的持续学习方法能够缓解玩具大小的数据集上的灾难性问题,但在将其应用于现实世界的问题时,仍有一些问题有待解决。最近,提出了基于掩码的快速学习方法(例如,piggyback \cite{mallya2018piggyback}),以通过以快速方式仅学习二进制元素掩码来解决这些问题,同时保持骨干模型固定。然而,二进制掩码对于新任务具有有限的建模能力。最近的工作\cite{hung2019compacting}提出了一种基于压缩-增长的方法(CPG),通过部分训练骨干模型来实现新任务的更好准确性,但具有更高的训练成本,这使得它不可能部署到流行的最先进的边缘/移动学习中。本研究的主要目标是在持续学习环境下同时实现快速、高精度的多任务适应。因此,出于动机,我们提出了一种新的训练方法称为\textit{kernel-wise Soft Mask}(KSM),它学习一个内核明智的混合二进制和实值软掩模为每个任务,同时使用相同的骨干模型。这种软掩码可以被视为二进制掩码和适当缩放的实值张量的叠加,其提供了更丰富的表示能力,而无需低级内核支持以满足低硬件开销的目标。我们在多个基准数据集上验证了KSM与最近最先进的方法(例如Piggyback,Packnet,CPG等),这在准确性和训练成本方面都显示出良好的改进。
英文摘要:
Deep Neural Networks (DNN) could forget the knowledge about earlier tasks when learning new tasks, and this is known as \textit{catastrophic forgetting}. While recent continual learning methods are capable of alleviating the catastrophic problem on toy-sized datasets, some issues still remain to be tackled when applying them in real-world problems. Recently, the fast mask-based learning method (e.g. piggyback \cite{mallya2018piggyback}) is proposed to address these issues by learning only a binary element-wise mask in a fast manner, while keeping the backbone model fixed. However, the binary mask has limited modeling capacity for new tasks. A more recent work \cite{hung2019compacting} proposes a compress-grow-based method (CPG) to achieve better accuracy for new tasks by partially training backbone model, but with order-higher training cost, which makes it infeasible to be deployed into popular state-of-the-art edge-/mobile-learning. The primary goal of this work is to simultaneously achieve fast and high-accuracy multi task adaption in continual learning setting. Thus motivated, we propose a new training method called \textit{kernel-wise Soft Mask} (KSM), which learns a kernel-wise hybrid binary and real-value soft mask for each task, while using the same backbone model. Such a soft mask can be viewed as a superposition of a binary mask and a properly scaled real-value tensor, which offers a richer representation capability without low-level kernel support to meet the objective of low hardware overhead. We validate KSM on multiple benchmark datasets against recent state-of-the-art methods (e.g. Piggyback, Packnet, CPG, etc.), which shows good improvement in both accuracy and training cost.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。