arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2212.06556cs.CVcs.CLcs.LG
收藏

用于微调视觉语言模型的本地化潜在更新

Localized Latent Updates for Fine-Tuning Vision-Language Models

Visual Computing Institute, RWTH Aachen University(亚琛工业大学视觉计算研究所)

Moritz Ibing, Isaak Lim, Leif Kobbelt

摘要:

尽管CLIP等大规模预训练的视觉语言模型在许多任务中表现出令人印象深刻的泛化能力,但仍然需要对其进行微调,以提高特定数据集的性能。当这样做时,期望更新模型是快速的,并且模型不会失去其对数据集之外的数据的能力,这通常是经典微调方法的情况。在这项工作中,我们提出了一个轻量级的适配器,只更新模型预测接近看到的数据点。我们在少样本学习的背景下证明了这种相对简单的方法的有效性和速度,其中我们在训练过程中看到和看不到的类上的结果与最先进的方法相当或有所改进。

英文摘要:

Although massive pre-trained vision-language models like CLIP show impressive generalization capabilities for many tasks, still it often remains necessary to fine-tune them for improved performance on specific datasets. When doing so, it is desirable that updating the model is fast and that the model does not lose its capabilities on data outside of the dataset, as is often the case with classical fine-tuning approaches. In this work we suggest a lightweight adapter, that only updates the models predictions close to seen datapoints. We demonstrate the effectiveness and speed of this relatively simple approach in the context of few-shot learning, where our results both on classes seen and unseen during training are comparable with or improve on the state of the art.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑