Mask-aware Text-to-Image Retrieval：Referring Expression Segmentation 与跨模态检索的结合
Mask-aware Text-to-Image Retrieval: Referring Expression Segmentation Meets Cross-modal Retrieval
AI总结：
本文提出统一文本到图像检索与指代分割的新任务 MaTIR，并设计两阶段框架，利用 SAM 2 和 Alpha-CLIP 离线生成掩码与区域嵌入以实现高效在线检索，再由 MLLM 重排序与定位，显著提升检索准确率与分割质量。
摘要：
文本到图像检索（TIR）旨在根据文本查询找到相关图像，但现有方法主要基于整图描述且缺乏可解释性。同时，referring expression segmentation（RES）能够基于自然语言描述实现精确的物体定位，但在大型图像集合上应用时计算成本高昂。为弥合这一差距，我们引入了 Mask-aware TIR（MaTIR），这是一种统一 TIR 和 RES 的新任务，既要求高效的图像搜索，又要求准确的物体分割。为解决该任务，我们提出了一个两阶段框架，第一阶段进行分割感知的图像检索，第二阶段利用多模态大语言模型（MLLM）进行重排序和物体定位。我们首先利用 SAM 2 生成物体掩码，并利用 Alpha-CLIP 离线提取区域级嵌入，从而实现有效且可扩展的在线检索。其次，使用 MLLM 优化检索排序并生成边界框，将其与分割掩码进行匹配。我们在 COCO 和 D$^3$ 数据集上评估了该方法，结果表明在检索准确率和分割质量上均显著优于以往方法。
英文摘要：
Text-to-image retrieval (TIR) aims to find relevant images based on a textual query, but existing approaches are primarily based on whole-image captions and lack interpretability. Meanwhile, referring expression segmentation (RES) enables precise object localization based on natural language descriptions but is computationally expensive when applied across large image collections. To bridge this gap, we introduce Mask-aware TIR (MaTIR), a new task that unifies TIR and RES, requiring both efficient image search and accurate object segmentation. To address this task, we propose a two-stage framework, comprising a first stage for segmentation-aware image retrieval and a second stage for reranking and object grounding with a multimodal large language model (MLLM). We leverage SAM 2 to generate object masks and Alpha-CLIP to extract region-level embeddings offline at first, enabling effective and scalable online retrieval. Secondly, MLLM is used to refine retrieval rankings and generate bounding boxes, which are matched to segmentation masks. We evaluate our approach on COCO and D$^3$ datasets, demonstrating significant improvements in both retrieval accuracy and segmentation quality over previous methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
