多模态大语言模型的模态不一致持续学习
Modality-Inconsistent Continual Learning of Multimodal Large Language Models
AI总结：
本文提出MICL，一种针对多模态大语言模型的持续学习场景，涉及不一致的模态和变化的任务类型。通过伪目标生成模块和基于指令的知识蒸馏，解决任务类型转移导致的遗忘问题，实验验证了方法的有效性。
摘要：
本文介绍Modality-Inconsistent Continual Learning (MICL)，一种针对多模态大语言模型的持续学习场景，涉及不一致的模态（图像、音频或视频）和变化的任务类型（描述或问答）。与现有视觉-only或模态递增设置不同，MICL结合模态和任务类型转移，两者均导致灾难性遗忘。为解决这些挑战，我们提出MoInCL，采用伪目标生成模块以减轻先前模态因任务类型转移导致的遗忘。它还结合基于指令的知识蒸馏，以在引入新模态时保持模型处理先前学习模态的能力。我们通过六个任务进行MICL基准测试，并进行实验验证我们MoInCL的有效性。实验结果突显了MoInCL的优越性，显示出显著优于代表性和最先进持续学习基线的改进。
英文摘要：
In this paper, we introduce Modality-Inconsistent Continual Learning (MICL), a new continual learning scenario for Multimodal Large Language Models (MLLMs) that involves tasks with inconsistent modalities (image, audio, or video) and varying task types (captioning or question-answering). Unlike existing vision-only or modality-incremental settings, MICL combines modality and task type shifts, both of which drive catastrophic forgetting. To address these challenges, we propose MoInCL, which employs a Pseudo Targets Generation Module to mitigate forgetting caused by task type shifts in previously seen modalities. It also incorporates Instruction-based Knowledge Distillation to preserve the model's ability to handle previously learned modalities when new ones are introduced. We benchmark MICL using a total of six tasks and conduct experiments to validate the effectiveness of our MoInCL. The experimental results highlight the superiority of MoInCL, showing significant improvements over representative and state-of-the-art continual learning baselines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
