CLAIR：CLIP 辅助的弱监督零样本跨域图像检索
CLAIR: CLIP-Aided Weakly Supervised Zero-Shot Cross-Domain Image Retrieval
AI总结：
本文提出 CLAIR 方法，利用 CLIP 相似度置信度精炼噪声伪标签，并通过多种对比损失、闭式跨域映射和可学习提示，提升弱监督零样本跨域图像检索性能。
摘要：
近年来，大型基础模型的发展使得为海量无标注数据生成伪标签变得容易，这使得无监督零样本跨域图像检索（UZS-CDIR）的相关性有所下降。因此，在本文中，我们将注意力转向由 CLIP 等大型基础模型生成的带有噪声伪标签的弱监督零样本跨域图像检索（WSZS-CDIR）。为此，我们提出 CLAIR，利用 CLIP 文本特征与图像特征之间的相似度计算置信度分数，从而对噪声伪标签进行精炼。此外，我们设计了实例间对比损失和簇间对比损失，将图像编码到类别感知的潜在空间中，并设计了一种跨域对比损失以缓解域差异。我们还以闭式解形式学习了一种新颖的跨域映射函数，仅使用 CLIP 文本嵌入将图像特征从一个域投影到另一个域，从而进一步对齐用于检索的图像特征。最后，我们通过引入一组额外的可学习提示来增强 CLAIR 的零样本泛化能力，以处理新类别。我们在 TUBerlin、Sketchy、Quickdraw 和 DomainNet 零样本数据集上进行了大量实验，结果表明，与现有的最先进方法相比，我们的 CLAIR 始终展现出优越的性能。
英文摘要：
The recent growth of large foundation models that can easily generate pseudo-labels for huge quantity of unlabeled data makes unsupervised Zero-Shot Cross-Domain Image Retrieval (UZS-CDIR) less relevant. In this paper, we therefore turn our attention to weakly supervised ZS-CDIR (WSZS-CDIR) with noisy pseudo labels generated by large foundation models such as CLIP. To this end, we propose CLAIR to refine the noisy pseudo-labels with a confidence score from the similarity between the CLIP text and image features. Furthermore, we design inter-instance and inter-cluster contrastive losses to encode images into a class-aware latent space, and an inter-domain contrastive loss to alleviate domain discrepancies. We also learn a novel cross-domain mapping function in closed-form, using only CLIP text embeddings to project image features from one domain to another, thereby further aligning the image features for retrieval. Finally, we enhance the zero-shot generalization ability of our CLAIR to handle novel categories by introducing an extra set of learnable prompts. Extensive experiments are carried out using TUBerlin, Sketchy, Quickdraw, and DomainNet zero-shot datasets, where our CLAIR consistently shows superior performance compared to existing state-of-the-art methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
