针对改写检索任务适配双编码器视觉-语言模型
Adapting Dual-encoder Vision-language Models for Paraphrased Retrieval
AI总结：
本文针对改写查询导致检索结果不一致的问题，提出从预训练语言模型出发适配双编码器视觉-语言模型，在保持零样本分类和检索精度的同时显著提升改写查询的排序相似性。
摘要：
近年来，双编码器视觉-语言模型（例如 CLIP）在文本到图像检索任务上取得了显著性能。然而，我们发现这些模型对于一对改写后的查询通常会产生截然不同的检索结果。这种行为可能使检索系统更难以预测，并导致用户感到沮丧。在本工作中，我们考虑改写文本到图像检索任务，即模型在给定一对改写查询时旨在返回相似的结果。首先，我们收集了一个改写图像描述数据集，以便对该任务进行定量评估。然后，我们假设现有双编码器模型的不良行为源于其文本编码器是在图像-句子对上训练的，缺乏捕捉改写查询之间语义相似性的能力。为了改进这一点，我们研究了多种训练双编码器模型的策略，这些策略从在大规模文本语料库上预训练的语言模型开始。与 CLIP 和 OpenCLIP 等公开的双编码器模型相比，采用我们最佳适配策略训练的模型在改写查询上取得了显著更高的排序相似性，同时保持了相似的零样本分类和检索准确率。
英文摘要：
In the recent years, the dual-encoder vision-language models (\eg CLIP) have achieved remarkable text-to-image retrieval performance. However, we discover that these models usually results in very different retrievals for a pair of paraphrased queries. Such behavior might render the retrieval system less predictable and lead to user frustration. In this work, we consider the task of paraphrased text-to-image retrieval where a model aims to return similar results given a pair of paraphrased queries. To start with, we collect a dataset of paraphrased image descriptions to facilitate quantitative evaluation for this task. We then hypothesize that the undesired behavior of existing dual-encoder model is due to their text towers which are trained on image-sentence pairs and lack the ability to capture the semantic similarity between paraphrased queries. To improve on this, we investigate multiple strategies for training a dual-encoder model starting from a language model pretrained on a large text corpus. Compared to public dual-encoder models such as CLIP and OpenCLIP, the model trained with our best adaptation strategy achieves a significantly higher ranking similarity for paraphrased queries while maintaining similar zero-shot classification and retrieval accuracy.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
