超越反遗忘：具有正向前向迁移的多模态持续指令微调
Beyond Anti-Forgetting: Multimodal Continual Instruction Tuning with Positive Forward Transfer
AI总结：
针对多模态持续指令微调中的负向前向迁移问题，提出Fwd-Prompt方法，通过将提示梯度投影至残差空间和预训练子空间，实现任务干扰最小化与预训练知识重用，在无需旧样本且更新更少参数下达到SOTA性能。
摘要：
多模态持续指令微调（MCIT）使多模态大语言模型（MLLMs）能够满足不断涌现的需求，而无需昂贵的重新训练。MCIT面临两大主要障碍：灾难性遗忘（旧知识被遗忘）和负向前向迁移（未来任务的性能下降）。尽管现有方法极大地缓解了灾难性遗忘，但它们仍受负向前向迁移的困扰。我们通过对输入嵌入执行奇异值分解（SVD），发现了不同输入嵌入中的巨大差异。这种差异导致模型为旧任务和预训练任务学习无关信息，从而引发灾难性遗忘和负向前向迁移。为了解决这些问题，我们提出了具有正向前向迁移的提示微调（Fwd-Prompt），这是一种基于提示的方法，它将提示梯度投影到残差空间以最小化任务间的干扰，并投影到预训练子空间以重用预训练知识。我们的实验表明，Fwd-Prompt在更新更少参数且不需要旧样本的情况下实现了最先进的性能。我们的研究阐明了在指令微调范式下持续使MLLMs适应新任务的潜力，并鼓励未来的研究探索MCIT。
英文摘要：
Multimodal Continual Instruction Tuning (MCIT) enables Multimodal Large Language Models (MLLMs) to meet continuously emerging requirements without expensive retraining. MCIT faces two major obstacles: catastrophic forgetting (where old knowledge is forgotten) and negative forward transfer (where the performance of future tasks is degraded). Although existing methods have greatly alleviated catastrophic forgetting, they still suffer from negative forward transfer. We discover a large discrepancy in different input embeddings by performing singular value decomposition (SVD) on input embeddings. This discrepancy results in the model learning irrelevant information for old and pre-trained tasks, leading to catastrophic forgetting and negative forward transfer. To address these issues, we propose Prompt Tuning with Positive Forward Transfer (Fwd-Prompt), a prompt-based method that projects the prompt gradient to the residual space to minimize interference between tasks and to the pre-trained subspace for reusing pre-trained knowledge. Our experiments demonstrate that Fwd-Prompt achieves state-of-the-art performance while updating fewer parameters and requiring no old samples. Our research illuminates the potential of continuously adapting MLLMs to new tasks under the instruction tuning paradigm and encourages future studies to explore MCIT.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
