通过混合专家适配器增强视觉-语言模型的持续学习
Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters
AI总结：
本文提出一种参数高效的持续学习框架，通过在预训练CLIP模型中动态集成混合专家适配器并结合分布判别自动选择器（DDAS）来减轻长期遗忘并保留零样本识别能力，在减少60%参数训练负担的同时优于现有最先进方法。
摘要：
持续学习能够赋予视觉-语言模型持续获取新知识的能力，而无需访问整个历史数据集。然而，由于（i）终身学习过程中的参数变化，以及（ii）与全模型微调相关的显著计算负担，缓解大规模模型中的性能下降并非易事。在本研究中，我们提出了一个参数高效的持续学习框架，以减轻视觉-语言模型在增量学习中的长期遗忘问题。我们的方法涉及预训练CLIP模型的动态扩展，通过集成混合专家适配器来响应新任务。为了保留视觉-语言模型的零样本识别能力，我们进一步引入了分布判别自动选择器（DDAS），该选择器自动将分布内和分布外输入分别路由到MoE适配器和原始CLIP。通过在各种设置下的广泛实验，我们提出的方法持续优于以前的最先进方法，同时将参数训练负担减少了60%。我们的代码位于https://github.com/JiazuoYu/MoE-Adapters4CL
英文摘要：
Continual learning can empower vision-language models to continuously acquire new knowledge, without the need for access to the entire historical dataset. However, mitigating the performance degradation in large-scale models is non-trivial due to (i) parameter shifts throughout lifelong learning and (ii) significant computational burdens associated with full-model tuning. In this work, we present a parameter-efficient continual learning framework to alleviate long-term forgetting in incremental learning with vision-language models. Our approach involves the dynamic expansion of a pre-trained CLIP model, through the integration of Mixture-of-Experts (MoE) adapters in response to new tasks. To preserve the zero-shot recognition capability of vision-language models, we further introduce a Distribution Discriminative Auto-Selector (DDAS) that automatically routes in-distribution and out-of-distribution inputs to the MoE Adapter and the original CLIP, respectively. Through extensive experiments across various settings, our proposed method consistently outperforms previous state-of-the-art approaches while concurrently reducing parameter training burdens by 60%. Our code locates at https://github.com/JiazuoYu/MoE-Adapters4CL
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
