MIRe：通过无融合模态交互增强多模态查询表征以实现多模态检索
MIRe: Enhancing Multimodal Queries Representation via Fusion-Free Modality Interaction for Multimodal Retrieval
AI总结：
针对多模态检索中现有方法存在的文本主导、忽略关键视觉信息的问题，提出无融合模态交互的MIRe检索框架，搭配构建的预训练数据集，在零样本多模态检索基准上取得优异性能，有效缓解文本主导问题。
摘要：
近期的多模态检索方法通过利用视觉-文本对齐的预训练策略，赋予了基于文本的检索器多模态能力。这类方法在对齐过程中通常直接融合两种模态以进行交叉参照，从而理解多模态查询。然而，现有方法因存在文本主导问题——过度依赖文本驱动信号——往往会忽略关键的视觉信息。本文提出MIRe，一种在对齐过程中无需融合文本特征即可实现模态交互的检索框架。我们的方法允许文本查询关注视觉嵌入，同时不会将文本驱动信号反馈回视觉表征。此外，我们通过将简洁的问答对转换为扩展段落，构建了一个面向多模态查询检索的预训练数据集。实验表明，我们的预训练策略显著增强了对多模态查询的理解，在零样本设置下的四个多模态检索基准上取得了优异性能。此外，我们的消融研究与分析明确验证了该框架在缓解文本主导问题方面的有效性。我们的代码已公开：https://github.com/yeongjoonJu/MIRe
英文摘要：
Recent multimodal retrieval methods have endowed text-based retrievers with multimodal capabilities by utilizing pre-training strategies for visual-text alignment. They often directly fuse the two modalities for cross-reference during the alignment to understand multimodal queries. However, existing methods often overlook crucial visual information due to a text-dominant issue, which overly depends on text-driven signals. In this paper, we introduce MIRe, a retrieval framework that achieves modality interaction without fusing textual features during the alignment. Our method allows the textual query to attend to visual embeddings while not feeding text-driven signals back into the visual representations. Additionally, we construct a pre-training dataset for multimodal query retrieval by transforming concise question-answer pairs into extended passages. Our experiments demonstrate that our pre-training strategy significantly enhances the understanding of multimodal queries, resulting in strong performance across four multimodal retrieval benchmarks under zero-shot settings. Moreover, our ablation studies and analyses explicitly verify the effectiveness of our framework in mitigating the text-dominant issue. Our code is publicly available: https://github.com/yeongjoonJu/MIRe
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
