FACap：一个用于细粒度组合图像检索的大规模时尚数据集
FACap: A Large-scale Fashion Dataset for Fine-grained Composed Image Retrieval
AI总结：
本文提出大规模自动构建的时尚领域组合图像检索数据集FACap，并基于BLIP-2提出结合轻量级适配器和多头匹配的FashionBLIP-2模型，显著提升了细粒度时尚图像检索性能。
摘要：
组合图像检索（CIR）任务是在给定参考图像和修改文本的情况下检索目标图像。最近的CIR方法利用大型预训练视觉语言模型（VLMs），并在颜色和纹理等通用领域概念上取得了良好性能。然而，它们在时尚等应用领域仍然表现不佳，因为时尚中使用的丰富多样词汇需要特定的细粒度视觉和语言理解。另一个困难是缺乏具有详细且相关标注的大规模时尚数据集，这是由于专家进行手动标注的成本昂贵。为了解决这些挑战，我们引入了FACap，这是一个大规模、自动构建的时尚领域CIR数据集。它利用网络获取的时尚图像以及由VLM和大型语言模型（LLM）驱动的两阶段标注流程，生成准确且详细的修改文本。然后，我们提出了一种新的CIR模型FashionBLIP-2，它在FACap上使用轻量级适配器和多头查询-候选匹配对通用领域BLIP-2模型进行微调，以更好地处理细粒度时尚特定信息。FashionBLIP-2在Fashion IQ基准和增强评估数据集enhFashionIQ上进行了有无额外微调的评估，利用我们的流程获取更高质量的标注。实验结果表明，FashionBLIP-2与FACap预训练的结合显著提升了模型在时尚CIR中的性能，特别是在使用细粒度修改文本进行检索时，证明了我们的数据集和方法在电子商务网站等高要求环境中的价值。代码可在https://fgxaos.github.io/facap-paper-website/ 获取。
英文摘要：
The composed image retrieval (CIR) task is to retrieve target images given a reference image and a modification text. Recent methods for CIR leverage large pretrained vision-language models (VLMs) and achieve good performance on general-domain concepts like color and texture. However, they still struggle with application domains like fashion, because the rich and diverse vocabulary used in fashion requires specific fine-grained vision and language understanding. An additional difficulty is the lack of large-scale fashion datasets with detailed and relevant annotations, due to the expensive cost of manual annotation by specialists. To address these challenges, we introduce FACap, a large-scale, automatically constructed fashion-domain CIR dataset. It leverages web-sourced fashion images and a two-stage annotation pipeline powered by a VLM and a large language model (LLM) to generate accurate and detailed modification texts. Then, we propose a new CIR model FashionBLIP-2, which fine-tunes the general-domain BLIP-2 model on FACap with lightweight adapters and multi-head query-candidate matching to better account for fine-grained fashion-specific information. FashionBLIP-2 is evaluated with and without additional fine-tuning on the Fashion IQ benchmark and the enhanced evaluation dataset enhFashionIQ, leveraging our pipeline to obtain higher-quality annotations. Experimental results show that the combination of FashionBLIP-2 and pretraining with FACap significantly improves the model's performance in fashion CIR especially for retrieval with fine-grained modification texts, demonstrating the value of our dataset and approach in a highly demanding environment such as e-commerce websites. Code is available at https://fgxaos.github.io/facap-paper-website/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
