arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2405.14726cs.CV
收藏

蒸馏视觉-语言预训练以实现高效跨模态检索

Distilling Vision-Language Pretraining for Efficient Cross-Modal Retrieval

Meta AI;Korea University(高丽大学);University of Central Florida(中佛罗里达大学)

Young Kyun Jang, Donghyun Kim, Ser-nam Lim

AI总结:

本文提出DCMQ方法,利用视觉-语言预训练模型作为教师蒸馏知识至哈希学生模型,结合NPC变换和PQG量化,显著提升跨模态检索性能。

摘要:

“学习哈希”是高效检索的一种实用解决方案,具有快速搜索速度和低存储成本。它广泛应用于各种应用,如图像-文本跨模态搜索。在本文中,我们探讨了利用强大的大型预训练模型(如视觉-语言预训练(VLP)模型)的普及来增强学习哈希性能的潜力。我们提出了一种名为跨模态量化蒸馏(DCMQ)的新方法,该方法利用VLP模型的丰富语义知识来改进哈希表示学习。具体来说,我们将VLP作为“教师”,将知识蒸馏到配备码本的“学生”哈希模型中。这一过程涉及用VLP的丰富语义替换由多热向量组成且缺乏语义的监督标签。最后,我们应用一种称为带配对一致性的归一化(NPC)的变换,以获得判别性的蒸馏目标。此外,我们引入了一种新的量化方法,即带Gumbel的乘积量化(PQG),该方法促进平衡的码本学习,从而提高检索性能。广泛的基准测试表明,DCMQ始终优于现有的监督跨模态哈希方法,展示了其巨大的潜力。

英文摘要:

``Learning to hash'' is a practical solution for efficient retrieval, offering fast search speed and low storage cost. It is widely applied in various applications, such as image-text cross-modal search. In this paper, we explore the potential of enhancing the performance of learning to hash with the proliferation of powerful large pre-trained models, such as Vision-Language Pre-training (VLP) models. We introduce a novel method named Distillation for Cross-Modal Quantization (DCMQ), which leverages the rich semantic knowledge of VLP models to improve hash representation learning. Specifically, we use the VLP as a `teacher' to distill knowledge into a `student' hashing model equipped with codebooks. This process involves the replacement of supervised labels, which are composed of multi-hot vectors and lack semantics, with the rich semantics of VLP. In the end, we apply a transformation termed Normalization with Paired Consistency (NPC) to achieve a discriminative target for distillation. Further, we introduce a new quantization method, Product Quantization with Gumbel (PQG) that promotes balanced codebook learning, thereby improving the retrieval performance. Extensive benchmark testing demonstrates that DCMQ consistently outperforms existing supervised cross-modal hashing approaches, showcasing its significant potential.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑