CL-MoE：基于双动量专家混合机制增强多模态大语言模型以应对持续视觉问答
CL-MoE: Enhancing Multimodal Large Language Model with Dual Momentum Mixture-of-Experts for Continual Visual Question Answering
AI总结：
针对多模态大语言模型在持续学习中的灾难性遗忘问题，提出双动量专家混合框架CL-MoE，通过双路由器策略与动态动量更新机制，在10个VQA任务上实现最先进性能。
摘要：
多模态大语言模型（MLLMs）因其在视觉语言任务（如视觉问答）中卓越的理解与生成能力，引起了研究者的广泛关注。然而，现实世界知识更新的快速节奏使得MLLMs的离线训练成本高昂，且在面对非平稳数据流时，MLLMs在学习过程中会遭遇灾难性遗忘。本文提出了一个基于MLLMs的双动量专家混合（CL-MoE）框架，用于持续视觉问答（VQA）。我们将MLLMs与持续学习相结合，以利用LLMs中丰富的常识知识。我们引入了双路由器专家混合（RMoE）策略，通过任务级和实例级路由器选择全局和局部专家，从而稳健地为最适合该任务的专家分配权重。接着，我们设计了动态动量专家混合（MMoE），根据专家与任务/实例之间的关系动态更新专家参数，使模型能够在吸收新知识的同时保持现有知识。大量实验结果表明，我们的方法在10个VQA任务上取得了最先进的性能，证明了该方法的有效性。
英文摘要：
Multimodal large language models (MLLMs) have garnered widespread attention from researchers due to their remarkable understanding and generation capabilities in visual language tasks (e.g., visual question answering). However, the rapid pace of knowledge updates in the real world makes offline training of MLLMs costly, and when faced with non-stationary data streams, MLLMs suffer from catastrophic forgetting during learning. In this paper, we propose an MLLMs-based dual momentum Mixture-of-Experts (CL-MoE) framework for continual visual question answering (VQA). We integrate MLLMs with continual learning to utilize the rich commonsense knowledge in LLMs. We introduce a Dual-Router MoE (RMoE) strategy to select the global and local experts using task-level and instance-level routers, to robustly assign weights to the experts most appropriate for the task. Then, we design a dynamic Momentum MoE (MMoE) to update the parameters of experts dynamically based on the relationships between the experts and tasks/instances, so that the model can absorb new knowledge while maintaining existing knowledge. The extensive experimental results indicate that our method achieves state-of-the-art performance on 10 VQA tasks, proving the effectiveness of our approach.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
