arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2303.10070cs.CV
收藏

采用通用参数高效微调的统一持续学习框架

A Unified Continual Learning Framework with General Parameter-Efficient Tuning

Peking University Shenzhen Graduate School(北京大学深圳研究生院);King Abdullah University of Science and Technology (KAUST)(阿卜杜拉国王科技大学(KAUST));Baidu Inc.(百度公司)

Qiankun Gao, Chen Zhao, Yifan Sun, Teng Xi, Gang Zhang, Bernard Ghanem, Jian Zhang

AI总结:

本文提出统一持续学习框架LAE,通过学习、动量积累与双专家集成兼容Adapter、LoRA、Prefix等通用PET方法,并在CIFAR100和ImageNet-R上取得最优性能。

摘要:

“预训练→下游适配”为持续学习(CL)带来了新的机遇与挑战。尽管近期CL领域的最优性能是通过参数高效微调(PET)适配范式取得的,但已有研究仅探索了prompt,使其应用局限于Transformer。本文将prompting定位为PET的一种实例,并提出一个采用通用PET的统一CL框架,称为Learning-Accumulation-Ensemble(LAE)。PET(例如使用Adapter、LoRA或Prefix)能够以更少的参数和资源使预训练模型适配下游任务。对于任意给定的PET方法,LAE框架通过三项新颖设计将其纳入CL:1)Learning(学习):预训练模型通过微调一个在线PET模块适配新任务,并采用本文提出的适配速度校准来对齐不同PET模块;2)Accumulation(积累):在线PET模块学到的任务特定知识通过动量更新积累到离线PET模块中;3)Ensemble(集成):推理阶段,分别利用在线/离线PET模块构建两个专家(分别受新任务/历史任务偏好),用于预测集成。实验表明,LAE可兼容一系列PET方法,并获得强大的CL能力。例如,采用Adapter PET的LAE在CIFAR100和ImageNet-R数据集上的最后增量准确率分别超过此前最优方法1.3%和3.6%。代码位于https://github.com/gqk/LAE。

英文摘要:

The "pre-training $\rightarrow$ downstream adaptation" presents both new opportunities and challenges for Continual Learning (CL). Although the recent state-of-the-art in CL is achieved through Parameter-Efficient-Tuning (PET) adaptation paradigm, only prompt has been explored, limiting its application to Transformers only. In this paper, we position prompting as one instantiation of PET, and propose a unified CL framework with general PET, dubbed as Learning-Accumulation-Ensemble (LAE). PET, e.g., using Adapter, LoRA, or Prefix, can adapt a pre-trained model to downstream tasks with fewer parameters and resources. Given a PET method, our LAE framework incorporates it for CL with three novel designs. 1) Learning: the pre-trained model adapts to the new task by tuning an online PET module, along with our adaptation speed calibration to align different PET modules, 2) Accumulation: the task-specific knowledge learned by the online PET module is accumulated into an offline PET module through momentum update, 3) Ensemble: During inference, we respectively construct two experts with online/offline PET modules (which are favored by the novel/historical tasks) for prediction ensemble. We show that LAE is compatible with a battery of PET methods and gains strong CL capability. For example, LAE with Adaptor PET surpasses the prior state-of-the-art by 1.3% and 3.6% in last-incremental accuracy on CIFAR100 and ImageNet-R datasets, respectively. Code is available at \url{https://github.com/gqk/LAE}.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑