X-Aligner：无需炫技的组合视觉检索
X-Aligner: Composed Visual Retrieval without the Bells and Whistles
AI总结：
X-Aligner通过结合视觉和文本查询，提升视频检索性能，采用多阶段训练和交叉注意力模块实现更优的多模态表示对齐。
摘要：
组合视频检索（CoVR）通过结合视觉和文本查询来促进视频检索。然而，现有CoVR框架通常在单一阶段融合多模态输入，仅在初始基线基础上取得微小提升。为此，我们提出了一种新的CoVR框架，利用视觉语言模型（VLMs）的表示能力。我们的框架包含一个新颖的交叉注意力模块X-Aligner，由多个交叉注意力层组成，逐步融合视觉和文本输入，并将它们的多模态表示与目标视频的表示对齐。为进一步增强多模态查询的表示，我们将视觉查询的描述作为额外输入。该框架分两阶段训练以保持预训练VLM的表示。第一阶段仅训练新引入的模块，第二阶段则进一步微调文本查询编码器。我们基于BLIP家族架构（即BLIP和BLIP-2）实现该框架，并在Webvid-CoVR数据集上进行训练。除了在Webvid-CoVR-Test上的领域内评估外，我们还在组合图像检索（CIR）数据集CIRCO和Fashion-IQ上进行了零样本评估。我们的框架在CoVR任务上取得了最先进的性能，在Webvid-CoVR-Test上达到Recall@1为63.93%，并在CIR任务上展示了强大的零样本泛化能力。
英文摘要：
Composed Video Retrieval (CoVR) facilitates video retrieval by combining visual and textual queries. However, existing CoVR frameworks typically fuse multimodal inputs in a single stage, achieving only marginal gains over initial baseline. To address this, we propose a novel CoVR framework that leverages the representational power of Vision Language Models (VLMs). Our framework incorporates a novel cross-attention module X-Aligner, composed of cross-attention layers that progressively fuse visual and textual inputs and align their multimodal representation with that of the target video. To further enhance the representation of the multimodal query, we incorporate the caption of the visual query as an additional input. The framework is trained in two stages to preserve the pretrained VLM representation. In the first stage, only the newly introduced module is trained, while in the second stage, the textual query encoder is also fine-tuned. We implement our framework on top of BLIP-family architecture, namely BLIP and BLIP-2, and train it on the Webvid-CoVR data set. In addition to in-domain evaluation on Webvid-CoVR-Test, we perform zero-shot evaluations on the Composed Image Retrieval (CIR) data sets CIRCO and Fashion-IQ. Our framework achieves state-of-the-art performance on CoVR obtaining a Recall@1 of 63.93% on Webvid-CoVR-Test, and demonstrates strong zero-shot generalization on CIR tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
