arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2305.04072cs.IR
收藏

基于语义感知对比学习与Transformer的关键词驱动多样化图像检索

Keyword-Based Diverse Image Retrieval by Semantics-aware Contrastive Learning and Transformer

Minyi Zhao, Jinpeng Wang, Dongliang Liao, Yiru Wang, Huanzhong Duan, Shuigeng Zhou

AI总结:

针对跨模态图像检索中相关性与多样性难以平衡的问题,提出CoLT方法,通过语义感知对比学习、Transformer token分类器及后处理算法提升检索多样性,在两个真实数据集上F1分数优于现有方法。

摘要:

在跨模态图像检索系统中,除相关性外,多样性是一项重要却研究较少的性能指标,对用户体验至关重要。现有的多样性感知图像检索方案要么对标准检索系统的原始检索结果进行显式后处理,要么尝试学习图像的多向量表示以刻画其多样语义。然而,这两种方法都难以很好地平衡相关性与多样性。一方面,标准检索系统通常偏向常见语义,训练过程中很少引入多样性感知正则化,因此通过后处理提升多样性的效果有限。另一方面,多向量表示方法无法保证学习到鲁棒的多重投影,导致不相关图像以及具有稀有或独特语义的图像可能被错误投影,进而降低top-k等典型算法生成结果的相关性与多样性。针对上述问题,本文提出一种名为CoLT的新方法,旨在生成更具代表性和鲁棒性的表示,以实现准确的图像分类。具体而言,CoLT首先通过语义感知对比学习增强现有一对一跨模态系统的初步表示,提取语义感知图像特征;随后构建基于Transformer的token分类器,将所有特征归入对应类别;最后设计后处理算法,从每个类别中检索图像以形成最终检索结果。在Div400和Div150Cred两个真实数据集上的大量实验表明,CoLT能有效提升多样性,整体性能优于现有方法(F1分数更高)。

英文摘要:

In addition to relevance, diversity is an important yet less studied performance metric of cross-modal image retrieval systems, which is critical to user experience. Existing solutions for diversity-aware image retrieval either explicitly post-process the raw retrieval results from standard retrieval systems or try to learn multi-vector representations of images to represent their diverse semantics. However, neither of them is good enough to balance relevance and diversity. On the one hand, standard retrieval systems are usually biased to common semantics and seldom exploit diversity-aware regularization in training, which makes it difficult to promote diversity by post-processing. On the other hand, multi-vector representation methods are not guaranteed to learn robust multiple projections. As a result, irrelevant images and images of rare or unique semantics may be projected inappropriately, which degrades the relevance and diversity of the results generated by some typical algorithms like top-k. To cope with these problems, this paper presents a new method called CoLT that tries to generate much more representative and robust representations for accurately classifying images. Specifically, CoLT first extracts semantics-aware image features by enhancing the preliminary representations of an existing one-to-one cross-modal system with semantics-aware contrastive learning. Then, a transformer-based token classifier is developed to subsume all the features into their corresponding categories. Finally, a post-processing algorithm is designed to retrieve images from each category to form the final retrieval result. Extensive experiments on two real-world datasets Div400 and Div150Cred show that CoLT can effectively boost diversity, and outperforms the existing methods as a whole (with a higher F1 score).

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑