对多模态大语言模型进行微调而不遗忘比你想象的更容易
Fine-tuning MLLMs Without Forgetting Is Easier Than You Think
AI总结：
本文通过实验框架展示，简单调整微调方法可缓解灾难性遗忘，提出数据混合训练策略解决特定任务过拟合问题，并证明该方法可扩展至持续学习。
摘要：
本文证明，对多模态大语言模型（MLLM）的微调配方进行简单调整足以缓解灾难性遗忘。在视觉问答任务中，我们设计了一个2x2实验框架来评估模型在分布内和分布外图像和文本输入下的性能。我们的结果表明，适当的正则化，如限制可训练参数数量或采用低学习率，能有效防止处理分布外图像时的遗忘。然而，我们发现当处理分布内图像和分布外文本时会出现一种不同的遗忘形式。我们将这种遗忘归因于任务特定的过拟合，并通过引入数据混合训练策略来解决这一问题，该策略结合了数据集和任务。最后，我们证明这种方法自然扩展到持续学习，优于现有方法，这些方法使用复杂的辅助机制。总的来说，我们的发现挑战了普遍假设，通过突出MLLM的内在鲁棒性，并提供实际指南，以在保持其通用能力的同时调整这些模型。
英文摘要：
The paper demonstrate that simple adjustments of the fine-tuning recipes of multimodal large language models (MLLM) are sufficient to mitigate catastrophic forgetting. On visual question answering, we design a 2x2 experimental framework to assess model performance across in-distribution and out-of-distribution image and text inputs. Our results show that appropriate regularization, such as constraining the number of trainable parameters or adopting a low learning rate, effectively prevents forgetting when dealing with out-of-distribution images. However, we uncover a distinct form of forgetting in settings with in-distribution images and out-of-distribution text. We attribute this forgetting as task-specific overfitting and address this issue by introducing a data-hybrid training strategy that combines datasets and tasks. Finally, we demonstrate that this approach naturally extends to continual learning, outperforming existing methods with complex auxiliary mechanisms. In general, our findings challenge the prevailing assumptions by highlighting the inherent robustness of MLLMs and providing practical guidelines for adapting them while preserving their general capabilities.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
