无视觉检索：用文本场景描述重新思考多模态搜索
Vision-Free Retrieval: Rethinking Multimodal Search with Textual Scene Descriptions
AI总结：
本文提出一种无视觉的单编码器文本到文本检索方法，利用VLLM生成图像描述替代视觉编码器，显著缩小模态差距并提升组合性，在多个基准上以0.3B参数模型取得最先进的零样本性能。
摘要：
以对比方式训练的视觉-语言模型（VLM），如CLIP，已成为学习判别性视觉-语言表示的标准方法。然而，这些模型往往表现出浅层的语言理解能力，呈现出词袋行为。这些局限性因其双编码器设计而进一步加剧，该设计会导致模态差距。此外，训练过程中对海量网络收集数据语料的依赖使得计算成本高昂，并带来显著的隐私问题。为了解决这些局限性，在本工作中，我们通过引入一种无视觉的单编码器检索流程，挑战了检索任务中视觉编码器的必要性。我们脱离传统的文本到图像检索范式，借助VLLM生成的结构化图像描述，迁移到文本到文本范式。我们证明，这种范式转变具有显著优势，包括大幅缩小模态差距、改善组合性，以及在短标题查询和长标题查询上均取得更好的性能，而所有这些仅需在两块GPU上进行数小时的校准即可实现。此外，用文本描述替代原始图像为检索提供了一种更有利于隐私保护的替代方案。为了进一步评估泛化能力并解决先前组合性基准的一些不足，我们发布了两个源自Flickr30k和COCO的基准，其中包含由短标题构成的多样化组合查询，我们将它们命名为subFlickr和subCOCO。我们的无视觉检索器能够匹敌并常常超越传统的多模态模型。重要的是，我们的方法在多个检索和组合性基准上取得了最先进的零样本性能，所用模型参数规模小至0.3B。代码可在以下地址获取：https://github.com/IoannaNti/LexiCLIP
英文摘要：
Contrastively-trained Vision-Language Models (VLMs), such as CLIP, have become the standard approach for learning discriminative vision-language representations. However, these models often exhibit shallow language understanding, manifesting bag-of-words behaviour. These limitations are reinforced by their dual-encoder design, which induces a modality gap. Additionally, the reliance on vast web-collected data corpora for training makes the process computationally expensive and introduces significant privacy concerns. To address these limitations, in this work, we challenge the necessity of vision encoders for retrieval tasks by introducing a vision-free, single-encoder retrieval pipeline. Departing from the traditional text-to-image retrieval paradigm, we migrate to a text-to-text paradigm with the assistance of VLLM-generated structured image descriptions. We demonstrate that this paradigm shift has significant advantages, including a substantial reduction of the modality gap, improved compositionality, and better performance on short and long caption queries, all attainable with only a few hours of calibration on two GPUs. Additionally, substituting raw images with textual descriptions introduces a more privacy-friendly alternative for retrieval. To further assess generalisation and address some of the shortcomings of prior compositionality benchmarks, we release two benchmarks derived from Flickr30k and COCO, containing diverse compositional queries made of short captions, which we coin subFlickr and subCOCO. Our vision-free retriever matches and often surpasses traditional multimodal models. Importantly, our approach achieves state-of-the-art zero-shot performance on multiple retrieval and compositionality benchmarks, with models as small as 0.3B parameters. Code is available at: https://github.com/IoannaNti/LexiCLIP
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
