arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2411.04892cs.CV
收藏

在视觉-语言模型的提示学习时代

In the Era of Prompt Learning with Vision-Language Models

LNMIIT(印度理工学院卢迪亚纳分校(LNMIIT))

Ankit Jha

AI总结:

针对视觉-语言模型在领域偏移下泛化能力不足的问题,提出StyLIP和AD-CLIP两种提示学习方法,分别用于领域泛化和无监督领域适应,通过解耦风格与内容并对比训练,在多个基准上超越现有方法,并展望了遥感语义分割中的类别发现。

摘要:

像CLIP这样的大规模基础模型在零样本泛化方面表现出色,但在面对领域偏移时却表现不佳,这限制了它们的适应性。在我们的工作中,我们引入了StyLIP,一种用于领域泛化(DG)的新颖的领域无关提示学习策略。StyLIP通过使用风格投影器学习领域特定的提示标记,并将其与内容特征相结合,从而解耦CLIP视觉编码器中的视觉风格和内容。该方法通过对比训练,实现了跨领域的无缝适应,在多个DG基准上超越了最先进的方法。此外,我们提出了AD-CLIP用于无监督领域适应(DA),利用CLIP冻结的视觉骨干网络,通过图像风格和内容特征学习领域不变的提示。通过在嵌入空间中结合熵最小化来对齐领域,AD-CLIP有效处理了领域偏移,即使仅当目标领域样本可用时也是如此。最后,我们概述了未来工作,即利用提示学习进行遥感语义分割中的类别发现,重点是在非结构化环境中识别新颖或稀有类别。这为在复杂、现实世界场景中构建更具适应性和泛化性的模型铺平了道路。

英文摘要:

Large-scale foundation models like CLIP have shown strong zero-shot generalization but struggle with domain shifts, limiting their adaptability. In our work, we introduce \textsc{StyLIP}, a novel domain-agnostic prompt learning strategy for Domain Generalization (DG). StyLIP disentangles visual style and content in CLIP`s vision encoder by using style projectors to learn domain-specific prompt tokens and combining them with content features. Trained contrastively, this approach enables seamless adaptation across domains, outperforming state-of-the-art methods on multiple DG benchmarks. Additionally, we propose AD-CLIP for unsupervised domain adaptation (DA), leveraging CLIP`s frozen vision backbone to learn domain-invariant prompts through image style and content features. By aligning domains in embedding space with entropy minimization, AD-CLIP effectively handles domain shifts, even when only target domain samples are available. Lastly, we outline future work on class discovery using prompt learning for semantic segmentation in remote sensing, focusing on identifying novel or rare classes in unstructured environments. This paves the way for more adaptive and generalizable models in complex, real-world scenarios.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑