arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.00979cs.CV

面向零样本草图图像检索的带辅助文本描述的跨模态注意力对齐网络

Cross-Modal Attention Alignment Network with Auxiliary Text Description for zero-shot sketch-based image retrieval

School of Computer and Electronic Information, Nanjing Normal University(南京师范大学计算机与电子信息学院)

Hanwen Su, Ge Song, Kai Huang, Jiyan Wang, Ming Yang

AI总结:

本文提出利用大语言模型为图像生成辅助文本描述,并通过跨模态注意力对齐网络实现零样本草图图像检索,在三个基准数据集上取得优于现有方法的性能。

摘要:

本文研究零样本草图图像检索(ZS-SBIR)问题。以往方法通常只在双模态设定下处理该问题,仅使用类别标签,甚至完全不涉及文本信息。然而,大规模预训练语言模型(LLMs)日益普及,并展现出从网络规模数据中学习到的丰富知识,这为我们整合集体文本信息提供了机会。我们的核心创新在于将文本数据用作图像的辅助信息,从而利用语言本身具备的零样本泛化能力。为此,我们提出一种面向零样本草图图像检索的带辅助文本描述的跨模态注意力对齐网络方法。该网络由三个部分组成:(i)描述生成模块,通过向LLM提示若干疑问句,为每个训练类别生成文本描述;(ii)特征提取模块,包含两个分别用于草图和图像数据的ViT,以及一个用于提取每个训练类别句子token的transformer;(iii)跨模态对齐模块,利用交叉注意力机制交换文本-草图和文本-图像的token特征,并在局部和全局层面对token进行对齐。在三个基准数据集上的大量实验表明,我们的性能优于当前最先进的ZS-SBIR方法。

英文摘要:

In this paper, we study the problem of zero-shot sketch-based image retrieval (ZS-SBIR). The prior methods tackle the problem in a two-modality setting with only category labels or even no textual information involved. However, the growing prevalence of Large-scale pre-trained Language Models (LLMs), which have demonstrated great knowledge learned from web-scale data, can provide us with an opportunity to conclude collective textual information. Our key innovation lies in the usage of text data as auxiliary information for images, thus leveraging the inherent zero-shot generalization ability that language offers. To this end, we propose an approach called Cross-Modal Attention Alignment Network with Auxiliary Text Description for zero-shot sketch-based image retrieval. The network consists of three components: (i) a Description Generation Module that generates textual descriptions for each training category by prompting an LLM with several interrogative sentences, (ii) a Feature Extraction Module that includes two ViTs for sketch and image data, a transformer for extracting tokens of sentences of each training category, finally (iii) a Cross-modal Alignment Module that exchanges the token features of both text-sketch and text-image using cross-attention mechanism, and align the tokens locally and globally. Extensive experiments on three benchmark datasets show our superior performances over the state-of-the-art ZS-SBIR methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑