arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2404.00710cs.CV
收藏

未知提示,唯一的空白:揭示CLIP在开放域泛化中的潜力

Unknown Prompt, the only Lacuna: Unveiling CLIP's Potential for Open Domain Generalization

Aisin Corporation(爱信公司);IIT Bombay(印度理工学院孟买分校);TU Munich(慕尼黑工业大学);IISER Thiruvananthapuram(特里凡得琅印度科学教育与研究学院);Deakin University(迪肯大学)

Mainak Singha, Ankit Jha, Shirsha Bose, Ashwin Nair, Moloud Abdar, Biplab Banerjee

AI总结:

本文提出ODG-CLIP框架,通过未知类别提示建模、视觉风格中心提示学习及提示空间知识注入,解决开放域泛化中的域与类别偏移问题,性能提升8%-16%。

摘要:

我们深入研究开放域泛化(ODG),其特点是训练的有标签源域与测试的无标签目标域之间存在域偏移和类别偏移。现有的ODG解决方案由于传统CNN骨干网络泛化能力受限,以及在缺乏先验知识时检测目标开放样本存在错误而面临局限。针对这些缺陷,我们提出了ODG-CLIP,利用视觉语言模型CLIP的语义能力。我们的框架带来了三项主要创新:首先,与现有范式不同,我们将ODG概念化为一个包含已知类别和新颖类别的多类分类挑战。我们方法的核心是建模一个专门用于检测未知类别样本的独特提示,为了训练该提示,我们采用一个易于获取的稳定扩散模型,优雅地为开放类生成代理图像。其次,为了获得面向域定制的分类(提示)权重,同时确保精度与简洁性的平衡,我们设计了一种新颖的视觉风格中心提示学习机制。最后,我们将从提示空间中获得的类别判别知识注入图像,以增强CLIP视觉嵌入的保真度。我们引入了一个新颖的目标,以保护这种注入的语义信息跨域的连续性,尤其是对于共享类别。通过在涵盖封闭集和开放集DG场景的多个数据集上进行严格测试,ODG-CLIP展现出明显的优势,持续超越同类方法,性能提升幅度在8%-16%之间。代码将在 https://github.com/mainaksingha01/ODG-CLIP 提供。

英文摘要:

We delve into Open Domain Generalization (ODG), marked by domain and category shifts between training's labeled source and testing's unlabeled target domains. Existing solutions to ODG face limitations due to constrained generalizations of traditional CNN backbones and errors in detecting target open samples in the absence of prior knowledge. Addressing these pitfalls, we introduce ODG-CLIP, harnessing the semantic prowess of the vision-language model, CLIP. Our framework brings forth three primary innovations: Firstly, distinct from prevailing paradigms, we conceptualize ODG as a multi-class classification challenge encompassing both known and novel categories. Central to our approach is modeling a unique prompt tailored for detecting unknown class samples, and to train this, we employ a readily accessible stable diffusion model, elegantly generating proxy images for the open class. Secondly, aiming for domain-tailored classification (prompt) weights while ensuring a balance of precision and simplicity, we devise a novel visual stylecentric prompt learning mechanism. Finally, we infuse images with class-discriminative knowledge derived from the prompt space to augment the fidelity of CLIP's visual embeddings. We introduce a novel objective to safeguard the continuity of this infused semantic intel across domains, especially for the shared classes. Through rigorous testing on diverse datasets, covering closed and open-set DG contexts, ODG-CLIP demonstrates clear supremacy, consistently outpacing peers with performance boosts between 8%-16%. Code will be available at https://github.com/mainaksingha01/ODG-CLIP.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑