通过实体接地实现组合式图像-文本匹配与检索
Compositional Image-Text Matching and Retrieval by Grounding Entities
AI总结：
本文针对CLIP缺乏实体接地和组合匹配能力的问题，提出免学习的零样本嵌入增强方法，通过开放词汇检测器定位实体与关系并加权组合子图像嵌入，在多个匹配与检索基准上取得显著提升。
摘要：
在大规模图像-文本对数据集上进行视觉-语言预训练，是当前视觉-语言模型的主要构建模块之一。通过额外训练，这些模型在视觉问答、图像描述和视觉常识推理等多种下游任务中表现出色。然而，像CLIP这样的预训练模型存在一个显著弱点，即无法执行实体接地以及组合式图像和文本匹配。在本工作中，我们提出了一种新颖的免学习零样本CLIP嵌入增强方法，该方法具有良好的组合特性。我们利用最先进的开放词汇检测器对物体实体和关系进行定位，计算其子图像嵌入，并动态调整基线全局图像嵌入。最终嵌入通过计算子图像嵌入的加权组合获得，随后用于与文本嵌入进行相似度计算，在Visual Genome和SVO Probes数据集上使图像-文本匹配准确率平均提升了1.5%。值得注意的是，增强后的嵌入展现出更优的检索性能，在Flickr30K和MS-COCO检索基准上取得了显著增益，将最先进的Recall@1分别提升了12%和0.4%。我们的代码可在https://github.com/madhukarreddyvongala/GroundingCLIP获取。
英文摘要：
Vision-language pretraining on large datasets of images-text pairs is one of the main building blocks of current Vision-Language Models. While with additional training, these models excel in various downstream tasks, including visual question answering, image captioning, and visual commonsense reasoning. However, a notable weakness of pretrained models like CLIP, is their inability to perform entity grounding and compositional image and text matching~\cite{Jiang2024ComCLIP, yang2023amc, Rajabi2023GroundedVSR, learninglocalizeCVPR24}. In this work we propose a novel learning-free zero-shot augmentation of CLIP embeddings that has favorable compositional properties. We compute separate embeddings of sub-images of object entities and relations that are localized by the state of the art open vocabulary detectors and dynamically adjust the baseline global image embedding. % The final embedding is obtained by computing a weighted combination of the sub-image embeddings. The resulting embedding is then utilized for similarity computation with text embedding, resulting in a average 1.5\% improvement in image-text matching accuracy on the Visual Genome and SVO Probes datasets~\cite{krishna2017visualgenome, svo}. Notably, the enhanced embeddings demonstrate superior retrieval performance, thus achieving significant gains on the Flickr30K and MS-COCO retrieval benchmarks~\cite{flickr30ke, mscoco}, improving the state-of-the-art Recall@1 by 12\% and 0.4\%, respectively. Our code is available at https://github.com/madhukarreddyvongala/GroundingCLIP.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
