混合专家遇见基于提示的持续学习
Mixture of Experts Meets Prompt-Based Continual Learning
AI总结：
本文从理论上揭示预训练模型注意力块的混合专家结构，将prefix tuning解释为新增任务专家，并提出非线性残差门控NoRGa以提升参数高效的持续学习性能。
摘要：
借助预训练模型的能力，基于提示的方法在有效防止灾难性遗忘方面相较于其他持续学习方案表现突出，即使其可学习参数极少且无需记忆缓冲区也是如此。尽管现有基于提示的持续学习方法擅长利用提示取得最先进性能，但它们往往缺乏对提示有效性的理论解释。本文开展理论分析，以揭示提示如何在持续学习中赋予这些优势，从而为提示设计提供新视角。我们首先表明，Vision Transformers等预训练模型的注意力块天然编码了一种特殊的混合专家架构，其特征是线性专家和二次门控分数函数。这一认识促使我们对prefix tuning提出新看法，将其重构为添加新的任务特定专家，从而启发设计出一种称为Non-linear Residual Gates（NoRGa）的新型门控机制。通过引入非线性激活和残差连接，NoRGa在保持参数效率的同时提升了持续学习性能。NoRGa的有效性在多种基准和预训练范式上得到理论与实验验证。我们的代码公开于https://github.com/Minhchuyentoancbn/MoE_PromptCL。
英文摘要：
Exploiting the power of pre-trained models, prompt-based approaches stand out compared to other continual learning solutions in effectively preventing catastrophic forgetting, even with very few learnable parameters and without the need for a memory buffer. While existing prompt-based continual learning methods excel in leveraging prompts for state-of-the-art performance, they often lack a theoretical explanation for the effectiveness of prompting. This paper conducts a theoretical analysis to unravel how prompts bestow such advantages in continual learning, thus offering a new perspective on prompt design. We first show that the attention block of pre-trained models like Vision Transformers inherently encodes a special mixture of experts architecture, characterized by linear experts and quadratic gating score functions. This realization drives us to provide a novel view on prefix tuning, reframing it as the addition of new task-specific experts, thereby inspiring the design of a novel gating mechanism termed Non-linear Residual Gates (NoRGa). Through the incorporation of non-linear activation and residual connection, NoRGa enhances continual learning performance while preserving parameter efficiency. The effectiveness of NoRGa is substantiated both theoretically and empirically across diverse benchmarks and pretraining paradigms. Our code is publicly available at https://github.com/Minhchuyentoancbn/MoE_PromptCL
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
