arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.22137cs.AI
收藏

CART:用于多模态持续指令微调的兼容路由

CART: Compatible Routing for Multimodal Continual Instruction Tuning

Nanjing University(南京大学);State Key Laboratory of Novel Software Technology, Nanjing University(南京大学新型软件技术国家重点实验室)

Ling-Xiang Zhang, Tao Hu, Da-Wei Zhou

AI总结:

针对多模态持续指令微调中投影器与语言模型非对称适配引发的灾难性遗忘问题,提出CART方法,通过协调双模块适配与专家合并实现最优性能。

摘要:

多模态大语言模型(MLLMs)通过指令微调实现了优异性能,但实际应用中常需从连续任务中持续获取新能力,即多模态持续指令微调(MCIT)。典型MLLMs采用投影器对齐视觉表示与语言模型,现有方法主要保护语言模型避免遗忘,同时跨任务持续更新投影器。这种非对称适配会导致投影器出现显著表示漂移,加剧投影器与语言模型的不匹配,进而加重灾难性遗忘。为解决该问题,本文提出兼容路由(CART),明确协调投影器与语言模型的适配。具体而言,CART为两个模块均维护任务特定专家,并通过加权混合组合它们;CART通过同时考虑专家相关性和跨模块兼容性来联合推断权重,从而促进两个模块间的协调适配。为进一步控制模型增长,本文引入相关性引导的专家合并,即合并每个模块内样本间相关性特征相似的专家。大量实验表明,CART取得了最优性能。

英文摘要:

Multimodal Large Language Models (MLLMs) achieve strong performance through instruction tuning, yet real-world applications often require them to continually acquire new capabilities from sequential tasks, i.e., Multimodal Continual Instruction Tuning (MCIT). In typical MLLMs, a projector is employed to align visual representations with the language model. However, existing methods primarily protect the language model from forgetting while continuously updating the projector across tasks. Such asymmetric adaptation can cause substantial representation drift in the projector, increasing the mismatch between the projector and language model and consequently exacerbating catastrophic forgetting. To address this issue, we propose Compatible Routing (CART), which explicitly coordinates the adaptation of the projector and language model. Specifically, CART maintains task-specific experts for both modules and combines them through weighted mixtures. CART jointly infers their weights by considering both expert relevance and cross-module compatibility, thereby promoting coordinated adaptation between the two modules. To further control model growth, we introduce relevance-guided expert consolidation, which merges experts exhibiting similar relevance profiles across samples within each module. Extensive experiments demonstrate that CART achieves state-of-the-art performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑