ContextBLIP：面向语言复杂描述对比图像检索的双重上下文对齐
ContextBLIP: Doubly Contextual Alignment for Contrastive Image Retrieval from Linguistically Complex Descriptions
AI总结：
针对语言复杂描述下的对比图像检索，提出ContextBLIP方法，通过上下文内和上下文间双重对齐策略，以极小参数量达到与GPT-4V相当的性能。
摘要：
从上下文描述进行图像检索（IRCD）旨在根据语言上复杂的文本，在一组最小对比候选中识别图像。尽管视觉语言模型（VLM）取得了成功，但它们在IRCD方面仍明显落后于人类表现。主要挑战在于对齐两种模态中的关键上下文线索，这些微妙线索隐藏在多张对比图像的微小区域中，并存在于文本描述的复杂语言中。这促使我们提出ContextBLIP，一种简单而有效的方法，它依赖于双重上下文对齐方案来处理具有挑战性的IRCD。具体来说，1）我们的模型包含一个多尺度适配器、一个匹配损失和一个文本引导的掩码损失。适配器学习捕捉细粒度的视觉线索。这两个损失为适配器提供迭代监督，逐步将单个图像的关键补丁突出到关键文本线索上。我们将这种方式称为上下文内对齐。2）然后，ContextBLIP进一步采用上下文间编码器来学习候选之间的依赖关系，促进文本与多张图像之间的对齐。我们将此步骤称为上下文间对齐。因此，隐藏在每个模态中的细微线索可以被有效对齐。在两个基准上的实验显示了我们的方法的优越性。我们观察到，尽管ContextBLIP涉及的参数数量约为GPT-4V的1/7500，但它可以产生与GPT-4V相当的结果。
英文摘要：
Image retrieval from contextual descriptions (IRCD) aims to identify an image within a set of minimally contrastive candidates based on linguistically complex text. Despite the success of VLMs, they still significantly lag behind human performance in IRCD. The main challenges lie in aligning key contextual cues in two modalities, where these subtle cues are concealed in tiny areas of multiple contrastive images and within the complex linguistics of textual descriptions. This motivates us to propose ContextBLIP, a simple yet effective method that relies on a doubly contextual alignment scheme for challenging IRCD. Specifically, 1) our model comprises a multi-scale adapter, a matching loss, and a text-guided masking loss. The adapter learns to capture fine-grained visual cues. The two losses enable iterative supervision for the adapter, gradually highlighting the focal patches of a single image to the key textual cues. We term such a way as intra-contextual alignment. 2) Then, ContextBLIP further employs an inter-context encoder to learn dependencies among candidates, facilitating alignment between the text to multiple images. We term this step as inter-contextual alignment. Consequently, the nuanced cues concealed in each modality can be effectively aligned. Experiments on two benchmarks show the superiority of our method. We observe that ContextBLIP can yield comparable results with GPT-4V, despite involving about 7,500 times fewer parameters.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
