arXiv 2503.042292025-03-06cs.CVcs.LG

合成数据是持续视觉语言模型的一份优雅GIFT

Synthetic Data is an Elegant GIFT for Continual Vision-Language Models

School of Computer Science, Wuhan University（武汉大学计算机学院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Wuhan AI Research（武汉人工智能研究院）；Taikang Center for Life and Medical Sciences, Wuhan University（武汉大学泰康生命医学中心）

Bin Wu, Wuxuan Shi, Jinqiao Wang, Mang Ye

备注：: This work is accepted by CVPR 2025. Modifications may be performed

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出GIFT方法，利用扩散模型生成合成数据重建预训练与下游任务数据，通过对比蒸馏和自适应权重整合克服视觉语言模型持续学习中的灾难性遗忘，实现更优的稳定性与可塑性平衡。

摘要：

预训练的视觉语言模型（VLMs）需要持续学习（CL）来高效更新知识并适应各种下游任务，而无需从头开始重新训练。然而，对于VLMs，除了从下游任务中学习到的知识发生丢失外，预训练知识在持续微调过程中也会遭到破坏。原始预训练数据的不可用性加剧了这一问题，导致VLM的泛化能力不断下降。在本文中，我们提出了GIFT，一种利用合成数据克服VLMs灾难性遗忘的新型持续微调方法。借助文本到图像合成的最新进展，我们采用预训练扩散模型来重建预训练数据和已学习的下游任务数据。通过这种方式，VLM可以在匹配的扩散生成图像和对应文本提示上进行蒸馏，从而重温以前的知识。利用合成图像-文本对在VLM特征空间中的广泛分布和高对齐度，我们提出了一种对比蒸馏损失以及图像-文本对齐约束。为了进一步对抗分布内过拟合，并在生成数据量有限的情况下增强蒸馏性能，我们引入了自适应权重 consolidation，利用这些合成图像-文本对的 Fisher 信息，实现了更好的稳定性-可塑性平衡。大量实验表明，我们的方法在各种设置下始终优于先前的最先进方法。

英文摘要：

Pre-trained Vision-Language Models (VLMs) require Continual Learning (CL) to efficiently update their knowledge and adapt to various downstream tasks without retraining from scratch. However, for VLMs, in addition to the loss of knowledge previously learned from downstream tasks, pre-training knowledge is also corrupted during continual fine-tuning. This issue is exacerbated by the unavailability of original pre-training data, leaving VLM's generalization ability degrading. In this paper, we propose GIFT, a novel continual fine-tuning approach that utilizes synthetic data to overcome catastrophic forgetting in VLMs. Taking advantage of recent advances in text-to-image synthesis, we employ a pre-trained diffusion model to recreate both pre-training and learned downstream task data. In this way, the VLM can revisit previous knowledge through distillation on matching diffusion-generated images and corresponding text prompts. Leveraging the broad distribution and high alignment between synthetic image-text pairs in VLM's feature space, we propose a contrastive distillation loss along with an image-text alignment constraint. To further combat in-distribution overfitting and enhance distillation performance with limited amount of generated data, we incorporate adaptive weight consolidation, utilizing Fisher information from these synthetic image-text pairs and achieving a better stability-plasticity balance. Extensive experiments demonstrate that our method consistently outperforms previous state-of-the-art approaches across various settings.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。