细粒度零样本组合图像检索与互补视觉-语义整合
Fine-Grained Zero-Shot Composed Image Retrieval with Complementary Visual-Semantic Integration
AI总结：
本文提出CVSI方法，通过互补视觉-语义整合提升细粒度零样本组合图像检索性能。
摘要：
零样本组合图像检索（ZS-CIR）是一个快速增长的领域，具有显著的实际应用价值，允许用户通过提供参考图像和描述所需修改的相对说明来检索目标图像。现有的ZS-CIR方法往往难以捕捉到细粒度的变化并有效整合视觉和语义信息。它们主要依赖于将多模态查询转换为单一文本的图像到文本模型，或者使用大型语言模型生成目标图像描述，这些方法通常无法捕捉到互补的视觉信息和完整的语义上下文。为了解决这些限制，我们提出了一种新的细粒度零样本组合图像检索方法，具有互补的视觉-语义整合（CVSI）。具体来说，CVSI利用三个关键组件：（1）视觉信息提取，不仅提取全局图像特征，还使用预训练的映射网络将图像转换为伪令牌，将其与修改文本和最有可能添加的对象结合起来。（2）语义信息提取，涉及使用预训练的描述生成模型为参考图像生成多个描述，然后利用LLM生成修改后的描述和最有可能添加的对象。（3）互补信息检索，整合从查询和数据库图像中提取的信息以检索目标图像，使系统能够高效地处理各种情况的检索查询。在三个公共数据集（例如CIRR、CIRCO和FashionIQ）上的广泛实验表明，CVSI显著优于现有最先进的方法。我们的代码可在https://github.com/yyc6631/CVSI上获得。
英文摘要：
Zero-shot composed image retrieval (ZS-CIR) is a rapidly growing area with significant practical applications, allowing users to retrieve a target image by providing a reference image and a relative caption describing the desired modifications. Existing ZS-CIR methods often struggle to capture fine-grained changes and integrate visual and semantic information effectively. They primarily rely on either transforming the multimodal query into a single text using image-to-text models or employing large language models for target image description generation, approaches that often fail to capture complementary visual information and complete semantic context. To address these limitations, we propose a novel Fine-Grained Zero-Shot Composed Image Retrieval method with Complementary Visual-Semantic Integration (CVSI). Specifically, CVSI leverages three key components: (1) Visual Information Extraction, which not only extracts global image features but also uses a pre-trained mapping network to convert the image into a pseudo token, combining it with the modification text and the objects most likely to be added. (2) Semantic Information Extraction, which involves using a pre-trained captioning model to generate multiple captions for the reference image, followed by leveraging an LLM to generate the modified captions and the objects most likely to be added. (3) Complementary Information Retrieval, which integrates information extracted from both the query and database images to retrieve the target image, enabling the system to efficiently handle retrieval queries in a variety of situations. Extensive experiments on three public datasets (e.g., CIRR, CIRCO, and FashionIQ) demonstrate that CVSI significantly outperforms existing state-of-the-art methods. Our code is available at https://github.com/yyc6631/CVSI.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
