基于生成数据集的正则化训练实现视觉-语言模型的仅名称迁移
Regularized Training with Generated Datasets for Name-Only Transfer of Vision-Language Models
AI总结：
本文针对仅名称迁移中生成图像与真实图像的域差距导致微调性能下降的问题，提出训练后权重空间集成与训练时特征多样性正则化两种方法，在多个数据集上取得最先进性能。
摘要：
近年来，文本到图像生成技术的进展激励研究者利用生成模型为感知模型定制数据集，这在真实数据稀缺的场景中尤为有价值。本研究旨在解决在生成数据集上微调视觉-语言模型（如 CLIP）所面临的挑战。具体而言，我们的目标是在无法获取任何真实图像的情况下，将视觉-语言模型微调为特定的分类模型，即所谓的仅名称迁移（name-only transfer）。然而，尽管生成图像具有高保真度，我们观察到，由于真实图像与生成图像之间存在域差距，直接使用生成数据集微调模型会导致显著的性能下降。为克服这一域差距，我们分别针对训练阶段和训练后阶段提出了两种正则化方法。首先，我们利用原始预训练视觉-语言模型中与域无关的知识，在训练后阶段将在生成数据集上微调得到的模型与原始预训练模型进行权重空间集成。其次，我们发现具有高特征多样性的微调模型在真实域中表现优异，这表明提高特征多样性可以防止模型学习生成域特有的知识。因此，我们在训练阶段引入额外的正则化来促进特征多样性。在多种分类数据集和多种文本到图像生成模型上的大量实验表明，我们的分析和正则化技术能够有效缓解长期被忽视的域差距问题，并使我们能够通过生成图像训练达到最先进的性能。代码可在 https://github.com/pmh9960/regft-for-gen 获取。
英文摘要：
Recent advancements in text-to-image generation have inspired researchers to generate datasets tailored for perception models using generative models, which prove particularly valuable in scenarios where real-world data is limited. In this study, our goal is to address the challenges when fine-tuning vision-language models (e.g., CLIP) on generated datasets. Specifically, we aim to fine-tune vision-language models to a specific classification model without access to any real images, also known as name-only transfer. However, despite the high fidelity of generated images, we observed a significant performance degradation when fine-tuning the model using the generated datasets due to the domain gap between real and generated images. To overcome the domain gap, we provide two regularization methods for training and post-training, respectively. First, we leverage the domain-agnostic knowledge from the original pre-trained vision-language model by conducting the weight-space ensemble of the fine-tuned model on the generated dataset with the original pre-trained model at the post-training. Secondly, we reveal that fine-tuned models with high feature diversity score high performance in the real domain, which indicates that increasing feature diversity prevents learning the generated domain-specific knowledge. Thus, we encourage feature diversity by providing additional regularization at training time. Extensive experiments on various classification datasets and various text-to-image generation models demonstrated that our analysis and regularization techniques effectively mitigate the domain gap, which has long been overlooked, and enable us to achieve state-of-the-art performance by training with generated images. Code is available at https://github.com/pmh9960/regft-for-gen
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
