SMoLoRA：探索并应对连续视觉指令微调中的双重灾难性遗忘
SMoLoRA: Exploring and Defying Dual Catastrophic Forgetting in Continual Visual Instruction Tuning
AI总结：
针对连续视觉指令微调中视觉理解与指令遵循双重灾难性遗忘问题，提出SMoLoRA框架，通过双路由模块实现领域专用适应，并构建新基准，有效缓解遗忘并提升泛化与鲁棒性。
摘要：
视觉指令微调（VIT）使多模态大语言模型（MLLMs）能够通过将各类视觉任务转化为基于语言的指令来有效处理它们。在此基础上，连续视觉指令微调（CVIT）扩展了MLLMs增量学习新任务的能力，以适应不断演进的功能需求。尽管先前的工作通过开发新基准和缓解灾难性遗忘的方法推进了CVIT，但这些努力大多遵循传统的连续学习范式，忽略了CVIT特有的挑战。我们在CVIT中识别出一种双重形式的灾难性遗忘：MLLMs不仅会遗忘先前学到的视觉理解能力，还会在获取新任务时出现指令遵循能力的下降。为解决此问题，我们引入了可分离低秩自适应混合（SMoLoRA）框架，该框架通过两个不同模块——一个用于视觉理解，另一个用于指令遵循——采用可分离路由。这种双路由设计实现了两个领域的专用适应，在防止遗忘的同时提升了性能。此外，我们提出了一个新的CVIT基准，它超越了现有基准，额外评估了模型对未见任务的泛化能力以及处理跨各种任务的多样指令的能力。大量实验表明，SMoLoRA在缓解双重遗忘、提升对未见任务的泛化能力以及确保遵循多样指令的鲁棒性方面均优于现有方法。代码可在 https://github.com/Minato-Zackie/SMoLoRA 获取。
英文摘要：
Visual instruction tuning (VIT) enables multimodal large language models (MLLMs) to effectively handle a wide range of vision tasks by framing them as language-based instructions. Building on this, continual visual instruction tuning (CVIT) extends the capability of MLLMs to incrementally learn new tasks, accommodating evolving functionalities. While prior work has advanced CVIT through the development of new benchmarks and approaches to mitigate catastrophic forgetting, these efforts largely follow traditional continual learning paradigms, neglecting the unique challenges specific to CVIT. We identify a dual form of catastrophic forgetting in CVIT, where MLLMs not only forget previously learned visual understanding but also experience a decline in instruction following abilities as they acquire new tasks. To address this, we introduce the Separable Mixture of Low-Rank Adaptation (SMoLoRA) framework, which employs separable routing through two distinct modules-one for visual understanding and another for instruction following. This dual-routing design enables specialized adaptation in both domains, preventing forgetting while improving performance. Furthermore, we propose a new CVIT benchmark that goes beyond existing benchmarks by additionally evaluating a model's ability to generalize to unseen tasks and handle diverse instructions across various tasks. Extensive experiments demonstrate that SMoLoRA outperforms existing methods in mitigating dual forgetting, improving generalization to unseen tasks, and ensuring robustness in following diverse instructions. Code is available at https://github.com/Minato-Zackie/SMoLoRA.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
