基于路由的多模态大语言模型持续学习
Routing-Based Continual Learning for Multimodal Large Language Models
AI总结：
本文提出一种基于路由的持续学习方法，用于多模态大语言模型，有效缓解灾难性遗忘并提升跨模态迁移性能，同时保持训练效率。
摘要：
多模态大语言模型（MLLMs）在持续学习中面临困难，通常在适应连续任务时出现灾难性遗忘。我们引入了一种基于路由的架构，能够在整合新能力的同时，稳健地保持基础知识。尽管多任务学习（MTL）提供了理论性能上限，但随着任务数量增加，其计算开销呈线性增长。相比之下，我们的方法保持了固定的数据和计算需求，无论任务序列长度如何。在从2B到8B参数的模型中，我们证明了我们的路由方法在性能上与MTL相当，同时保持了顺序微调的训练效率。除了缓解遗忘外，我们还观察到令牌级路由促进了跨模态迁移，利用一种模态的知识来增强另一种模态的性能。消融研究证实了该方法的可扩展性：即使在大型专家池中，路由仍保持稳健，并有效利用任务相关性。最后，我们显示我们的方法具有良好的可扩展性，较大模型相比完全专门化的微调表现出最小的退化。
英文摘要：
Multimodal Large Language Models (MLLMs) struggle with continual learning, often suffering from catastrophic forgetting when adapting to sequential tasks. We introduce a routing-based architecture that integrates new capabilities while robustly preserving foundational knowledge. While Multi-Task Learning (MTL) offers a theoretical performance upper bound, it incurs a linearly scaling computational overhead as the number of tasks increases. In contrast, our method maintains fixed data and compute requirements regardless of the task sequence length. Across models ranging from 2B to 8B parameters, we demonstrate that our routing approach performs on par with MTL while retaining the training efficiency of sequential fine-tuning. Beyond merely mitigating forgetting, we observe that token-level routing facilitates cross-modal transfer, leveraging knowledge from one modality to bolster performance in another. Ablation studies confirm the approach's scalability: routing remains robust even with large expert pools and effectively capitalizes on task relatedness. Finally, we show that our method scales favorably, with larger models exhibiting minimal degradation compared to fully specialized fine-tuning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
