arXiv 2505.185942025-05-24cs.CVcs.IR

EvdCLIP：利用大语言模型生成的实体视觉描述改进视觉-语言检索

EvdCLIP: Improving Vision-Language Retrieval with Entity Visual Descriptions from Large Language Models

Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Peng Cheng Laboratory（鹏城实验室）；School of Computer Science & Tech, Huazhong University of Science and Technology（华中科技大学计算机科学与技术学院）；College of Computer Science and Software Engineering, Shenzhen University（深圳大学计算机与软件学院）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）；Hong Kong University of Science and Technology（香港科技大学）

GuangHao Meng, Sunan He, Jinpeng Wang, Tao Dai, Letian Zhang, Jieming Zhu, Qing Li, Gang Wang, Rui Zhang, Yong Jiang

备注：: 9 pages, 6 figures

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出EvdCLIP，利用大语言模型生成实体视觉描述（EVDs）增强查询，并设计可训练的EVD感知重写器（EaRW）降低噪声，在图像-文本检索基准上验证了其优越性。

摘要：

视觉-语言检索（VLR）在学术界和工业界都引起了广泛关注，它涉及使用文本（或图像）作为查询来检索相应的图像（或文本）。然而，现有方法往往忽略了实体丰富的视觉语义知识，从而导致错误的检索结果。为了解决这个问题，我们提出了实体视觉描述增强的CLIP（EvdCLIP），旨在利用实体的视觉知识来丰富查询。具体来说，由于人类通过视觉线索来识别实体，我们使用大语言模型（LLM）生成实体视觉描述（EVDs）作为对齐线索，以补充文本数据。这些EVDs随后被整合到原始查询中，以创建视觉丰富、EVD增强的查询。此外，认识到EVD增强的查询可能引入噪声或低质量的扩展，我们开发了一种新颖的、可训练的EVD感知重写器（EaRW），用于视觉-语言检索任务。EaRW利用EVD知识和语言模型的生成能力来有效地重写查询。通过我们专门的训练策略，EaRW可以生成高质量、低噪声的EVD增强查询。在图像-文本检索基准上的大量定量和定性实验验证了EvdCLIP在视觉-语言检索任务上的优越性。

英文摘要：

Vision-language retrieval (VLR) has attracted significant attention in both academia and industry, which involves using text (or images) as queries to retrieve corresponding images (or text). However, existing methods often neglect the rich visual semantics knowledge of entities, thus leading to incorrect retrieval results. To address this problem, we propose the Entity Visual Description enhanced CLIP (EvdCLIP), designed to leverage the visual knowledge of entities to enrich queries. Specifically, since humans recognize entities through visual cues, we employ a large language model (LLM) to generate Entity Visual Descriptions (EVDs) as alignment cues to complement textual data. These EVDs are then integrated into raw queries to create visually-rich, EVD-enhanced queries. Furthermore, recognizing that EVD-enhanced queries may introduce noise or low-quality expansions, we develop a novel, trainable EVD-aware Rewriter (EaRW) for vision-language retrieval tasks. EaRW utilizes EVD knowledge and the generative capabilities of the language model to effectively rewrite queries. With our specialized training strategy, EaRW can generate high-quality and low-noise EVD-enhanced queries. Extensive quantitative and qualitative experiments on image-text retrieval benchmarks validate the superiority of EvdCLIP on vision-language retrieval tasks.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。