先融合再校准：面向多模态大语言模型的简单高效模态增量持续学习方法
Merge then Realign: Simple and Effective Modality-Incremental Continual Learning for Multimodal LLMs
AI总结：
针对多模态大语言模型模态增量持续学习中存在的遗忘与组件错位问题，提出无需修改架构的“先融合再校准”范式，实现了几乎无损的扩展性能。
摘要：
多模态大语言模型（Multimodal Large Language Models, MLLMs）的最新进展提升了其通用性，因为它们整合了越来越多的模态。考虑到训练MLLMs的高昂成本，复用现有模型并通过模态增量持续学习（Modality-incremental Continual Learning, MCL）将其扩展到更多模态是一种高效的方式。目前MCL的探索尚处于早期阶段。在本研究中，我们深入探究了MCL中性能下降的原因。我们发现，其性能下降不仅源于传统持续学习中存在的遗忘问题，还源于模态无关组件与模态特定组件之间的错位问题。为此，我们提出了一种极为简洁的MCL范式，名为“MErge then ReAlign”（先融合再校准，简称MERA），以同时解决遗忘和错位问题。MERA无需引入大量模型参数开销，也无需修改模型架构，因此易于部署，在MLLM社区中具有很高的复用性。大量实验证明了MERA的出色性能：当扩展到四种模态时，其反向相对增益平均达到99.84%，实现了几乎无损的MCL性能。我们的发现凸显了MCL中的错位问题。更广泛地说，本研究展示了如何在持续学习过程中调整MLLMs的不同组件。
英文摘要：
Recent advances in Multimodal Large Language Models (MLLMs) have enhanced their versatility as they integrate a growing number of modalities. Considering the heavy cost of training MLLMs, it is efficient to reuse the existing ones and extend them to more modalities through Modality-incremental Continual Learning (MCL). The exploration of MCL is in its early stages. In this work, we dive into the causes of performance degradation in MCL. We uncover that it suffers not only from forgetting as in traditional continual learning, but also from misalignment between the modality-agnostic and modality-specific components. To this end, we propose an elegantly simple MCL paradigm called "MErge then ReAlign" (MERA) to address both forgetting and misalignment. MERA avoids introducing heavy model budgets or modifying model architectures, hence is easy to deploy and highly reusable in the MLLM community. Extensive experiments demonstrate the impressive performance of MERA, holding an average of 99.84\% Backward Relative Gain when extending to four modalities, achieving nearly lossless MCL performance. Our findings underscore the misalignment issue in MCL. More broadly, our work showcases how to adjust different components of MLLMs during continual learning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
