面向持续多模态指令调优的课程LoRA专家动态混合方法
Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuning
AI总结：
针对多模态大语言模型持续指令调优中的架构冲突与模态不平衡问题，提出参数预算下动态演进架构的D-MoLE方法，通过动态层级专家分配与模态间持续课程策略，较最优基线平均提升15%，为首项从架构视角开展的MLLM持续学习研究。
摘要：
持续多模态指令调优对于让多模态大语言模型（Multimodal Large Language Models, MLLMs）适配不断演进的任务至关重要。然而，现有大多数方法采用固定架构，受限于静态的模型容量，难以适配新任务。我们提出在参数预算下演进架构以实现动态任务适配，这一方向尚未被探索，且存在两大挑战：1）任务架构冲突，即不同任务需要不同的层级适配方式；2）模态不平衡，即不同任务对各模态的依赖程度不均，会导致更新失衡。为应对这些挑战，我们提出了一种新颖的动态课程LoRA专家混合（Dynamic Mixture of Curriculum LoRA Experts, D-MoLE）方法，该方法能在受控参数预算下自动演进MLLM的架构，在持续适配新任务的同时保留已学知识。具体而言，我们提出了动态层级专家分配器，可跨层自动分配LoRA专家以解决架构冲突，并逐层路由指令以促进专家间的知识共享。随后，我们提出了基于梯度的模态间持续课程策略，根据任务内各模态的难度调整MLLM中各模块的更新比例，以缓解模态不平衡问题。大量实验表明，D-MoLE的性能显著优于现有最优基线，较最优基线实现了15%的平均提升。据我们所知，这是首个从架构视角研究MLLM持续学习的工作。
英文摘要：
Continual multimodal instruction tuning is crucial for adapting Multimodal Large Language Models (MLLMs) to evolving tasks. However, most existing methods adopt a fixed architecture, struggling with adapting to new tasks due to static model capacity. We propose to evolve the architecture under parameter budgets for dynamic task adaptation, which remains unexplored and imposes two challenges: 1) task architecture conflict, where different tasks require varying layer-wise adaptations, and 2) modality imbalance, where different tasks rely unevenly on modalities, leading to unbalanced updates. To address these challenges, we propose a novel Dynamic Mixture of Curriculum LoRA Experts (D-MoLE) method, which automatically evolves MLLM's architecture with controlled parameter budgets to continually adapt to new tasks while retaining previously learned knowledge. Specifically, we propose a dynamic layer-wise expert allocator, which automatically allocates LoRA experts across layers to resolve architecture conflicts, and routes instructions layer-wisely to facilitate knowledge sharing among experts. Then, we propose a gradient-based inter-modal continual curriculum, which adjusts the update ratio of each module in MLLM based on the difficulty of each modality within the task to alleviate the modality imbalance problem. Extensive experiments show that D-MoLE significantly outperforms state-of-the-art baselines, achieving a 15% average improvement over the best baseline. To the best of our knowledge, this is the first study of continual learning for MLLMs from an architectural perspective.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
