HiDe-PET:基于参数高效调优分层分解的持续学习
HiDe-PET: Continual Learning via Hierarchical Decomposition of Parameter-Efficient Tuning
AI总结:
针对预训练模型(PTMs)和参数高效调优(PET)在持续学习(CL)中的次优性能问题,提出统一框架HiDe-PET,通过分层分解CL目标并整合任务知识,在理论和实验上均超越现有基线。
摘要:
预训练模型(PTMs)的部署极大推动了持续学习(CL)领域的发展,实现了积极的知识迁移并增强了对灾难性遗忘的抵御能力。为了在任务序列到达时维持这些优势,一个有前景的方向是冻结预训练骨干网络,同时采用参数高效调优(PET)技术指导表示学习。尽管基于提示的PET在CL中很流行,但在我们对不同PTMs和目标任务的评估中,其经验设计常导致次优性能。为此,我们提出一个统一的CL框架,结合PTMs和PET,提供理论与经验上的进展。我们首先对预训练背景下的CL目标进行深入理论分析,将其分解为任务内预测、任务身份推断和任务自适应预测三个层次组件。然后提出分层分解PET(HiDe-PET),这是一种创新方法,通过主流PET技术整合任务特定和共享知识,并高效恢复预训练表示,显式优化分解后的目标。利用该框架,我们深入研究了实现策略、PET技术和PET架构的不同影响,以及在显著分布变化中的自适应知识积累。最后,在各种CL场景中,我们的方法在多种近期强基线方法上表现出显著优越的性能。
英文摘要:
The deployment of pre-trained models (PTMs) has greatly advanced the field of continual learning (CL), enabling positive knowledge transfer and resilience to catastrophic forgetting. To sustain these advantages for sequentially arriving tasks, a promising direction involves keeping the pre-trained backbone frozen while employing parameter-efficient tuning (PET) techniques to instruct representation learning. Despite the popularity of Prompt-based PET for CL, its empirical design often leads to sub-optimal performance in our evaluation of different PTMs and target tasks. To this end, we propose a unified framework for CL with PTMs and PET that provides both theoretical and empirical advancements. We first perform an in-depth theoretical analysis of the CL objective in a pre-training context, decomposing it into hierarchical components namely within-task prediction, task-identity inference and task-adaptive prediction. We then present Hierarchical Decomposition PET (HiDe-PET), an innovative approach that explicitly optimizes the decomposed objective through incorporating task-specific and task-shared knowledge via mainstream PET techniques along with efficient recovery of pre-trained representations. Leveraging this framework, we delve into the distinct impacts of implementation strategy, PET technique and PET architecture, as well as adaptive knowledge accumulation amidst pronounced distribution changes. Finally, across various CL scenarios, our approach demonstrates remarkably superior performance over a broad spectrum of recent strong baselines.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。