利用纯图像数据扩展提示指令驱动的零样本组合图像检索
Scaling Prompt Instructed Zero Shot Composed Image Retrieval with Image-Only Data
AI总结：
针对组合图像检索（CIR）三元组标注成本高的问题，提出用大语言模型基于纯图像生成训练数据，结合嵌入改写架构构建InstructCIR模型，在零样本CIR任务上性能超SOTA，且随生成数据量增加性能逼近监督基线。
摘要：
组合图像检索（Composed Image Retrieval，CIR）是指检索与参考图像加文本描述相匹配的图像的任务，其中文本以自然语言描述对参考图像的修改。传统上，为CIR设计的模型依赖包含参考图像、改写文本和目标图像的三元组数据。然而，整理此类三元组数据通常需要人工干预，导致成本高得令人望而却步。这一挑战阻碍了CIR模型训练的可扩展性，即便有大量未标记数据可用。随着基础模型的最新进展，我们倡导转变CIR训练范式，即大型语言模型（LLM）可以高效替代人工标注。具体而言，我们证明了大型图像描述生成模型和语言模型能够仅依靠未标注图像集合高效生成CIR所需数据。此外，我们提出了一种嵌入改写架构，可有效融合图像和文本模态。我们的模型名为InstructCIR，在CIRR和FashionIQ数据集上的零样本组合图像检索任务中优于现有最优方法。进一步地，我们证明通过增加生成数据的数量，我们的零样本模型性能可逐步接近有监督基线的性能。
英文摘要：
Composed Image Retrieval (CIR) is the task of retrieving images matching a reference image augmented with a text, where the text describes changes to the reference image in natural language. Traditionally, models designed for CIR have relied on triplet data containing a reference image, reformulation text, and a target image. However, curating such triplet data often necessitates human intervention, leading to prohibitive costs. This challenge has hindered the scalability of CIR model training even with the availability of abundant unlabeled data. With the recent advances in foundational models, we advocate a shift in the CIR training paradigm where human annotations can be efficiently replaced by large language models (LLMs). Specifically, we demonstrate the capability of large captioning and language models in efficiently generating data for CIR only relying on unannotated image collections. Additionally, we introduce an embedding reformulation architecture that effectively combines image and text modalities. Our model, named InstructCIR, outperforms state-of-the-art methods in zero-shot composed image retrieval on CIRR and FashionIQ datasets. Furthermore, we demonstrate that by increasing the amount of generated data, our zero-shot model gets closer to the performance of supervised baselines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
