MERGETUNE: 视觉-语言模型的持续微调
MERGETUNE: Continued Fine-Tuning of Vision-Language Models
AI总结：
MERGETUNE通过持续微调和线性模式连接，恢复视觉-语言模型在微调中丢失的预训练知识，提升基础-新样本泛化和稳健微调性能。
摘要：
对视觉-语言模型（VLMs）如CLIP进行微调时，常常会导致预训练知识的灾难性遗忘。先前的工作主要旨在在适应过程中缓解遗忘；然而，这种遗忘在这一过程中往往仍然不可避免。我们介绍了一种新的范式，持续微调（CFT），旨在在零样本模型已经适应之后恢复预训练知识。我们提出了一种简单且模型无关的CFT策略（命名为MERGETUNE），该策略由线性模式连接（LMC）引导，可以在不改变架构的情况下应用于现有的微调模型。给定一个微调后的模型，我们继续微调其可训练参数（例如，软提示或线性头部），以搜索一个持续模型，该模型具有两条低损失路径连接到零样本（例如，CLIP）和微调后的（例如，CoOp）解决方案。通过利用损失景观的几何特性，持续模型隐式地合并了这两种解决方案，恢复在微调后的模型中丢失的预训练知识。一个挑战是，普通的LMC约束需要从预训练任务中进行数据回放。我们通过一个二阶替代方法近似了该约束，消除了对大规模数据回放的需求。实验表明，MERGETUNE在基础-新样本泛化上将CoOp的调和均值提高了+5.6%，而无需增加参数。在稳健微调评估中，MERGETUNE的LMC合并模型在较低的推理成本下超越了集成基线，当与零样本模型集成时，实现了进一步的增益和最先进的结果。我们的代码可在https://github.com/Surrey-UP-Lab/MERGETUNE上获得。
英文摘要：
Fine-tuning vision-language models (VLMs) such as CLIP often leads to catastrophic forgetting of pretrained knowledge. Prior work primarily aims to mitigate forgetting during adaptation; however, forgetting often remains inevitable during this process. We introduce a novel paradigm, continued fine-tuning (CFT), which seeks to recover pretrained knowledge after a zero-shot model has already been adapted. We propose a simple, model-agnostic CFT strategy (named MERGETUNE) guided by linear mode connectivity (LMC), which can be applied post hoc to existing fine-tuned models without requiring architectural changes. Given a fine-tuned model, we continue fine-tuning its trainable parameters (e.g., soft prompts or linear heads) to search for a continued model which has two low-loss paths to the zero-shot (e.g., CLIP) and the fine-tuned (e.g., CoOp) solutions. By exploiting the geometry of the loss landscape, the continued model implicitly merges the two solutions, restoring pretrained knowledge lost in the fine-tuned counterpart. A challenge is that the vanilla LMC constraint requires data replay from the pretraining task. We approximate this constraint for the zero-shot model via a second-order surrogate, eliminating the need for large-scale data replay. Experiments show that MERGETUNE improves the harmonic mean of CoOp by +5.6% on base-novel generalisation without adding parameters. On robust fine-tuning evaluations, the LMC-merged model from MERGETUNE surpasses ensemble baselines with lower inference cost, achieving further gains and state-of-the-art results when ensembled with the zero-shot model. Our code is available at https://github.com/Surrey-UP-Lab/MERGETUNE.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
