大型多模态模型的持续指令调优
Continual Instruction Tuning for Large Multimodal Models
AI总结：
本文研究大型多模态模型在持续指令调优中的灾难性遗忘问题，建立首个基准，并提出基于任务相似性的正则化与模型扩展方法以持续提升性能。
摘要：
指令调优目前已成为一种广泛采用的方法，用于使大型多模态模型（LMMs）对齐人类意图。它统一了视觉-语言任务的数据格式，从而支持多任务联合训练。然而，视觉-语言任务在实践中不断被创建。与其在新任务到来时总是重新训练LMMs，持续学习为模型持续且高效地利用不断演变的数据提供了灵活性。本研究旨在探索以下两个问题：1）在持续指令调优中，LMMs是否仍会遭受灾难性遗忘？2）现有的三类持续学习方法是否仍适用于LMMs的持续指令调优？我们开展了广泛的研究来回答上述问题。首先，我们建立了该设置下的第一个基准，并揭示出在对LMMs进行持续指令调优时仍会观察到灾难性遗忘。然而，多任务联合指令调优能够促进模型的持续学习能力并缓解遗忘。其次，我们将经典的持续学习方法集成并适配到我们的场景中，证明了数据回放和模型扩展策略在不同场景下的有效性。相比之下，基于正则化的方法仅在已经对多个任务进行过联合指令调优的模型上表现良好。第三，我们深入研究了视觉-语言任务对之间的相关性和遗忘动态，并提出了基于任务相似性信息的正则化和模型扩展方法，用于LMMs的持续指令调优。实验结果表明，我们的方法能够持续提升模型性能。
英文摘要：
Instruction tuning is now a widely adopted approach to aligning large multimodal models (LMMs) to follow human intent. It unifies the data format of vision-language tasks, enabling multi-task joint training. However, vision-language tasks are constantly being created in practice. Instead of always re-training LMMs when new tasks arrive, continual learning offers flexibility for models to continually and efficiently exploit the evolving data. This work aims to explore the following two questions: 1) Do LMMs still suffer from catastrophic forgetting in continual instruction tuning? 2) Are the existing three classes of continual learning methods still applicable to the continual instruction tuning of LMMs? An extensive study is conducted to address the above questions. First, we establish the first benchmark in this setting and reveal that catastrophic forgetting is still observed when continually instruction-tuning LMMs. However, the multi-task joint instruction tuning can facilitate the model's continual learning ability and mitigate forgetting. Second, we integrate and adapt classic continual learning methods to our context, demonstrating the efficacy of data replay and model expansion strategies across diverse scenarios. In contrast, regularization-based methods only perform well on models that have been jointly instruction-tuned on multiple tasks. Third, we delve into the correlation and forgetting dynamics between vision-language task pairs and propose task-similarity-informed regularization and model expansion methods for continual instruction tuning of LMMs. Experimental results show that our approach consistently boosts the model's performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
