arXiv 2412.043782025-05-08cs.CVcs.AI

VladVA：LVLM 的判别式微调

VladVA: Discriminative Fine-tuning of LVLMs

Samsung AI Cambridge（三星AI剑桥）；Technical University of Iasi（伊阿西技术大学）；Queen Mary University of London（伦敦女王学院）

Yassine Ouali, Adrian Bulat, Alexandros Xenos, Anestis Zaganidis, Ioannis Maniadis Metaxas, Brais Martinez, Georgios Tzimiropoulos

备注：: Published at CVPR 2025

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出 VladVA，通过对比损失与下一 token 预测损失联合训练，并利用软提示和 LoRA 将生成式 LVLM 转换为判别式模型，在图像-文本检索和组合性上显著超越同规模 CLIP 类模型。

摘要：

以 CLIP 为代表的对比训练视觉-语言模型（VLM）已成为判别式视觉-语言表示学习的事实标准方法。然而，这些模型的语言理解能力有限，常常表现出“词袋”行为。与此同时，将视觉编码器与 LLM 相结合的大型视觉-语言模型（LVLM）已被证明能够进行细致的视觉-语言推理，但其自回归特性使其不太适合判别式任务。在本工作中，我们提出结合“两全其美”的方案：一种用于 LVLM 判别式微调的新训练方法，能够带来强大的判别能力和组合能力。本质上，我们的方法将生成式 LVLM 转换为判别式 LVLM，释放其强大的图像-文本判别能力，同时增强语言理解能力。我们的贡献包括：（1）一个精心设计的训练/优化框架，利用可变长度和粒度的图像-文本对，通过对比损失和下一 token 预测损失共同训练模型。同时进行了消融研究，以证明我们框架各组成部分的必要性；（2）一种参数高效的自适应方法，结合了软提示和 LoRA 适配器；（3）相比同等规模的最先进 CLIP 类模型取得显著改进，包括在标准图像-文本检索基准上的提升以及在组合性方面的显著增益。

英文摘要：

Contrastively-trained Vision-Language Models (VLMs) like CLIP have become the de facto approach for discriminative vision-language representation learning. However, these models have limited language understanding, often exhibiting a "bag of words" behavior. At the same time, Large Vision-Language Models (LVLMs), which combine vision encoders with LLMs, have been shown to be capable of detailed vision-language reasoning, yet their autoregressive nature renders them less suitable for discriminative tasks. In this work, we propose to combine "the best of both worlds": a new training approach for discriminative fine-tuning of LVLMs that results in strong discriminative and compositional capabilities. Essentially, our approach converts a generative LVLM into a discriminative one, unlocking its capability for powerful image-text discrimination combined with enhanced language understanding. Our contributions include (1) a carefully designed training/optimization framework that utilizes image-text pairs of variable length and granularity for training the model with both contrastive and next-token prediction losses. This is accompanied by ablation studies that justify the necessity of our framework's components; (2) a parameter-efficient adaptation method using a combination of soft prompting and LoRA adapters; (3) significant improvements over state-of-the-art CLIP-like models of similar size, including standard image-text retrieval benchmarks and notable gains in compositionality.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。