arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.12048cs.CL
收藏

Model Tailor:缓解多模态大语言模型中的灾难性遗忘

Model Tailor: Mitigating Catastrophic Forgetting in Multi-modal Large Language Models

Zhejiang University(浙江大学);Tencent Youtu Lab(腾讯优图实验室)

Didi Zhu, Zhongyi Sun, Zexi Li, Tao Shen, Ke Yan, Shouhong Ding, Kun Kuang, Chao Wu

AI总结:

本文提出Model Tailor方法,通过保留预训练参数并替换少量微调参数,结合稀疏掩码与补偿机制,有效缓解多模态大语言模型微调中的灾难性遗忘,在保持原始任务性能的同时适应新任务。

摘要:

灾难性遗忘是微调多模态大语言模型(MLLMs)时面临的关键挑战,在未见任务上提升性能往往会导致在原始任务上的性能显著下降。本文对MLLMs中的灾难性遗忘进行了全面分析,并提出了一种名为Model Tailor的训练后调整方法。该方法主要保留预训练参数,同时替换少量(≤10%)的微调参数,在原始任务上维持相对于预训练约99%的有效性,并在新任务上达到标准微调约97%的效果。具体而言,我们基于融合显著性分析与敏感性分析的策略,推导出一个稀疏掩码来识别“模型补丁”。随后,引入补偿机制来“装饰补丁”,从而增强模型在目标任务和原始任务上的表现。此外,该方法可适应多任务场景。通过在InstructBLIP和LLaVA-1.5上针对图像描述和视觉问答任务开展的大量实验,我们的方法在展现显著任务适应性的同时,保留了固有的预训练能力。

英文摘要:

Catastrophic forgetting emerges as a critical challenge when fine-tuning multi-modal large language models (MLLMs), where improving performance on unseen tasks often leads to a significant performance drop on the original tasks. This paper presents a comprehensive analysis of catastrophic forgetting in MLLMs and introduces a post-training adjustment method called Model Tailor. Our method primarily preserves the pre-trained parameters while replacing a small number ($\leq$ 10\%) of fine-tuned parameters, maintaining $\sim$ 99\% effectiveness on original tasks versus pre-training, and achieving $\sim$ 97\% on new tasks compared to standard fine-tuning. Specifically, we derive a sparse mask to identify the "model patch", based on a fusion strategy that integrates salience and sensitivity analysis. Subsequently, a compensation mechanism is introduced to "decorate the patch", enhancing the model's performance on both target and original tasks. Additionally, our method is adaptable to multi-task scenarios. Through extensive experiments on InstructBLIP and LLaVA-1.5 in both image captioning and visual question answering tasks, our approach demonstrates significant task adaptability while preserving inherent pre-trained capabilities.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑