CoRe-MoE：用于持续多模态指令调优的紧凑可混合混合专家模型
CoRe-MoE: Compact Reusable MoE for Continual Multimodal Instruction Tuning
浏览论文内容
中文总结 AI 辅助
针对持续多模态指令调优中 LoRA-MoE 参数开销大的问题，提出 CoRe-MoE 框架，通过复用方向基减少参数，在 MLLM 上性能提升达 5.90 点且参数仅为顺序 LoRA 的 1%。
中文摘要 AI 辅助
持续多模态指令调优要求多模态大语言模型依次获取新任务能力，同时保留先前学习的知识。LoRA-MoE 提供了一种有前景的解决方案，它引入了基于专家的容量，但重复学习和维护完整 LoRA 专家会导致大量参数开销。这引发了一个自然问题：对于每个新任务，完整的专家扩展是否必要？为回答该问题，我们分析了特定任务 LoRA 更新的奇异值分解（SVD），观察到它们在输入侧和输出侧 LoRA 方向子空间存在大量重叠，且特定任务的适应主要由这些子空间上的轻量级坐标捕获。受此观察启发，我们提出 CoRe-MoE，一种用于参数高效持续多模态指令调优的紧凑可混合混合专家（MoE）框架。CoRe-MoE 从初始专家库中提取可复用的输入侧和输出侧方向基，对于后续任务，仅训练紧凑坐标专家以及特定任务的低秩路由器。在两个代表性多模态大语言模型（MLLM）上的实验表明，CoRe-MoE 相较于最强的竞争基线，最终平均性能提升了多达 5.90 个点，同时后续任务所需的可训练参数不到顺序 LoRA 的 1%。代码公开可获取，链接为 this https URL。
英文摘要
Continual multimodal instruction tuning requires multimodal large language models to acquire new task abilities sequentially while preserving previously learned knowledge. LoRA-MoE provides a promising solution by introducing expert-based capacity, but repeatedly learning and maintaining full LoRA experts leads to substantial parameter overhead. This raises a natural question: is full expert expansion necessary for every new task? To answer it, we analyze the SVD of task-specific LoRA updates and observe substantial overlap in their input- and output-side LoRA direction subspaces, with task-specific adaptation largely captured by lightweight coordinates over these subspaces. Motivated by this observation, we propose CoRe-MoE, a Compact Reusable MoE framework for parameter-efficient continual multimodal instruction tuning. CoRe-MoE extracts reusable input- and output-side direction bases from an initial expert bank, and for subsequent tasks trains only compact coordinate experts together with task-specific low-rank routers. Experiments on two representative MLLMs show that CoRe-MoE improves final average performance over the strongest competing baseline by up to 5.90 points, while using less than 1% of the trainable parameters required by sequential LoRA for later tasks. The code is publicly available at https://github.com/runzezz/CoRe-MoE.
发表机构
- Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）
- School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）
机构由 AI 辅助整理，请以论文原文为准。
