arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2310.09291cs.CV
收藏

通过语言视觉实现免训练的组合式图像检索

Vision-by-Language for Training-Free Compositional Image Retrieval

Tübingen AI Center(蒂宾根人工智能中心);University of Tübingen(蒂宾根大学);University of Trento(特伦托大学);Helmholtz Munich(慕尼黑亥姆霍兹中心);Technical University of Munich(慕尼黑工业大学)

Shyamgopal Karthik, Karsten Roth, Massimiliano Mancini, Zeynep Akata

AI总结:

本文提出免训练的CIReVL方法,通过生成式VLM描述参考图像并由LLM按文本修改重组描述,再借助CLIP检索,实现零样本组合式图像检索,在多个基准上取得有竞争力甚至超越有监督方法的结果,并具备可解释性和可扩展性。

摘要:

给定一张图像和一个目标修改(例如一张埃菲尔铁塔的图像以及文本“没有人且在夜间”),组合式图像检索(Compositional Image Retrieval, CIR)旨在从数据库中检索相关的目标图像。虽然有监督方法依赖于标注三元组(即查询图像、文本修改和目标图像),但这种方式成本高昂;近期研究通过使用大规模视觉-语言模型(VLMs)执行零样本组合式图像检索(Zero-Shot CIR, ZS-CIR)来规避这一需求。然而,ZS-CIR中最先进的方法仍然需要在大量图像-文本对上训练任务特定的定制模型。在本工作中,我们提出以免训练的方式解决CIR,通过我们的“通过语言视觉的组合式图像检索”(Compositional Image Retrieval through Vision-by-Language, CIReVL),这是一个简单、人类可理解且可扩展的流程,能够有效地将大规模VLMs与大语言模型(LLMs)重新组合。通过使用预训练的生成式VLM为参考图像生成描述,并要求LLM基于文本目标修改重新组合该描述,以便随后通过例如CLIP进行检索,我们实现了模块化的语言推理。在四个ZS-CIR基准上,我们发现其性能具有竞争力,部分达到最先进水平——优于有监督方法。此外,CIReVL的模块化特性提供了无需重新训练的简单可扩展性,使我们既能研究ZS-CIR的缩放规律和瓶颈,又能轻松将部分结果扩展到超过先前报告结果的两倍以上。最后,我们表明CIReVL通过在语言领域中以模块化方式组合图像和文本,使CIR对人类可理解,从而使其具备可干预性,允许事后重新对齐失败案例。代码将在被接收后发布。

英文摘要:

Given an image and a target modification (e.g an image of the Eiffel tower and the text "without people and at night-time"), Compositional Image Retrieval (CIR) aims to retrieve the relevant target image in a database. While supervised approaches rely on annotating triplets that is costly (i.e. query image, textual modification, and target image), recent research sidesteps this need by using large-scale vision-language models (VLMs), performing Zero-Shot CIR (ZS-CIR). However, state-of-the-art approaches in ZS-CIR still require training task-specific, customized models over large amounts of image-text pairs. In this work, we propose to tackle CIR in a training-free manner via our Compositional Image Retrieval through Vision-by-Language (CIReVL), a simple, yet human-understandable and scalable pipeline that effectively recombines large-scale VLMs with large language models (LLMs). By captioning the reference image using a pre-trained generative VLM and asking a LLM to recompose the caption based on the textual target modification for subsequent retrieval via e.g. CLIP, we achieve modular language reasoning. In four ZS-CIR benchmarks, we find competitive, in-part state-of-the-art performance - improving over supervised methods. Moreover, the modularity of CIReVL offers simple scalability without re-training, allowing us to both investigate scaling laws and bottlenecks for ZS-CIR while easily scaling up to in parts more than double of previously reported results. Finally, we show that CIReVL makes CIR human-understandable by composing image and text in a modular fashion in the language domain, thereby making it intervenable, allowing to post-hoc re-align failure cases. Code will be released upon acceptance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑