类别级文本-图像检索的改进：利用扩散模型和视觉编码器弥合域差距
Category-level Text-to-Image Retrieval Improved: Bridging the Domain Gap with Diffusion Models and Vision Encoders
AI总结：
针对CLIP等视觉语言模型在类别级文本-图像检索中存在的模态差距问题，提出结合扩散模型生成视觉查询、视觉编码器计算相似度及聚合网络融合多模态信息的两步法，显著提升了检索性能。
摘要：
本研究探讨针对指定或描述某一语义类别的查询的文本-图像检索任务。尽管CLIP等视觉语言模型（VLM）提供了直接的开放词汇解决方案，但它们将文本和图像映射到表示空间中相距较远的区域，限制了检索性能。为弥合这一模态差距，我们提出一种两步法：首先，使用生成式扩散模型将文本查询转换为视觉查询；随后，通过视觉模型计算图像间相似度。此外，我们引入聚合网络，将多张生成图像整合为单一向量表示，并融合两种查询模态的相似度得分。该方法结合了视觉编码器、VLM和文本生成图像模型的前沿进展。大量评估实验表明，其性能始终优于仅依赖文本查询的检索方法。源代码可访问：https://github.com/faixan-khan/cletir
英文摘要：
This work explores text-to-image retrieval for queries that specify or describe a semantic category. While vision-and-language models (VLMs) like CLIP offer a straightforward open-vocabulary solution, they map text and images to distant regions in the representation space, limiting retrieval performance. To bridge this modality gap, we propose a two-step approach. First, we transform the text query into a visual query using a generative diffusion model. Then, we estimate image-to-image similarity with a vision model. Additionally, we introduce an aggregation network that combines multiple generated images into a single vector representation and fuses similarity scores across both query modalities. Our approach leverages advancements in vision encoders, VLMs, and text-to-image generation models. Extensive evaluations show that it consistently outperforms retrieval methods relying solely on text queries. Source code is available at: https://github.com/faixan-khan/cletir
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
