在推理阶段添加简单结构可提升视觉-语言组合性
Adding simple structure at inference improves Vision-Language Compositionality
AI总结：
针对双编码器视觉-语言模型组合性不足的问题，提出在推理阶段通过分割图像、匹配文本片段与图像块并聚合相似度的方法，无需训练即可稳定提升模型图文检索的组合性表现，尤其在属性-对象绑定任务中效果显著。
摘要：
CLIP等双编码器视觉-语言模型（VLM）被广泛应用于图文检索任务，但这类模型在组合性方面存在缺陷，表现出类似词袋的行为，限制了其检索性能。目前已有诸多训练方法被提出以提升这类模型的视觉-语言组合能力，相比之下，推理阶段技术受到的关注较少。本文提出在推理阶段添加简单结构：给定一张图像和一段说明文字，首先将图像分割为多个更小的图像块，其次提取包含对象、属性和关系的文本片段，接着利用VLM找到与文本片段匹配度最高的图像块以获得匹配对，最后聚合各匹配对的个体相似度，计算得到最终的图文相似度。我们基于多种主流双编码器VLM，在受控数据集和自然数据集上评估了该方法的视觉-语言组合性表现。结果表明，该方法无需任何训练即可持续提升所评估VLM的性能，彰显了推理阶段技术的潜力；其中在受控数据集上的属性-对象绑定任务中效果尤为突出。通过大量分析我们还发现：处理图像块是实现性能提升的核心要素，同时明确了可进一步优化推理阶段方法的具体方向。
英文摘要：
Dual encoder Vision-Language Models (VLM) such as CLIP are widely used for image-text retrieval tasks. However, those models struggle with compositionality, showing a bag-of-words-like behavior that limits their retrieval performance. Many different training approaches have been proposed to improve the vision-language compositionality capabilities of those models. In comparison, inference-time techniques have received little attention. In this paper, we propose to add simple structure at inference, where, given an image and a caption: i) we divide the image into different smaller crops, ii) we extract text segments, capturing objects, attributes and relations, iii) using a VLM, we find the image crops that better align with text segments obtaining matches, and iv) we compute the final image-text similarity aggregating the individual similarities of the matches. Based on various popular dual encoder VLMs, we evaluate our approach in controlled and natural datasets for VL compositionality. We find that our approach consistently improves the performance of evaluated VLMs without any training, which shows the potential of inference-time techniques. The results are especially good for attribute-object binding as shown in the controlled dataset. As a result of an extensive analysis: i) we show that processing image crops is actually essential for the observed gains in performance, and ii) we identify specific areas to further improve inference-time approaches.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
