arXiv 2309.094962024-01-02cs.CVcs.AI

基于CLIP的文本行人检索协同知识转移

CLIP-based Synergistic Knowledge Transfer for Text-based Person Retrieval

Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Peng Cheng Laboratory（鹏城实验室）；School of ECE, Peking University（北京大学电子与计算机工程学院）

Yating Liu, Yaowei Li, Zimo Liu, Wenming Yang, Yaowei Wang, Qingmin Liao

备注：: ICASSP2024(accepted). minor typos revision compared to version 1 in arxiv

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出基于CLIP的协同知识转移方法CSKT用于文本行人检索，通过双向提示转移模块和双适配器转移机制实现输入输出两侧知识协同，在仅训练7.4%参数下于三个基准数据集上超越现有方法。

摘要：

文本行人检索（TPR）旨在根据文本查询检索目标人物图像。主要挑战在于弥合视觉和语言模态之间的巨大差距，特别是在处理有限的大规模数据集时。本文提出一种基于CLIP的协同知识转移（CSKT）方法用于TPR。具体而言，为探索CLIP在输入侧的知识，我们首先提出双向提示转移（BPT）模块，由文本到图像和图像到文本的双向提示及耦合投影构成。其次，设计双适配器转移（DAT）以在视觉和语言的多头注意力（MHA）输出侧转移知识。这种协同双向协作机制促进了早期特征融合，并高效利用CLIP的现有知识。当训练参数仅占整个模型的7.4%时，CSKT在三个基准数据集上超越了最先进方法，展示了其卓越的效率、有效性和泛化能力。

英文摘要：

Text-based Person Retrieval (TPR) aims to retrieve the target person images given a textual query. The primary challenge lies in bridging the substantial gap between vision and language modalities, especially when dealing with limited large-scale datasets. In this paper, we introduce a CLIP-based Synergistic Knowledge Transfer (CSKT) approach for TPR. Specifically, to explore the CLIP's knowledge on input side, we first propose a Bidirectional Prompts Transferring (BPT) module constructed by text-to-image and image-to-text bidirectional prompts and coupling projections. Secondly, Dual Adapters Transferring (DAT) is designed to transfer knowledge on output side of Multi-Head Attention (MHA) in vision and language. This synergistic two-way collaborative mechanism promotes the early-stage feature fusion and efficiently exploits the existing knowledge of CLIP. CSKT outperforms the state-of-the-art approaches across three benchmark datasets when the training parameters merely account for 7.4% of the entire model, demonstrating its remarkable efficiency, effectiveness and generalization.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。