arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.01400cs.CV
收藏

GalLoP:为视觉-语言模型学习全局与局部提示

GalLoP: Learning Global and Local Prompts for Vision-Language Models

Conservatoire national des arts et métiers(法国国立工艺学院);Univ. Gustave Eiffel(古斯塔夫·埃菲尔大学);Sorbonne Université(索邦大学)

Marc Lafon, Elias Ramzi, Clément Rambour, Nicolas Audebert, Nicolas Thome

AI总结:

针对视觉-语言模型提示学习在精度与鲁棒性间的权衡问题,提出GalLoP方法,通过全局与局部视觉特征学习多样化提示,结合稀疏策略、提示dropout和多尺度策略,在11个数据集少样本任务中精度更优,且域泛化与OOD检测鲁棒性强。

摘要:

提示学习已被广泛用于高效适配视觉-语言模型(VLMs)(如CLIP)以实现少样本图像分类。尽管取得成功,但多数提示学习方法在分类精度与鲁棒性间存在权衡,例如在域泛化或分布外(OOD)检测任务中。本研究提出全局-局部提示(GalLoP),这是一种新的提示学习方法,可利用全局和局部视觉特征学习多个多样化提示。局部提示的训练依赖具有增强视觉-文本对齐的局部特征;为仅聚焦相关特征,这种局部对齐与局部特征选择的稀疏策略相结合。我们通过新的“提示 dropout”技术和局部提示的多尺度策略,确保提示集的多样性。在不同少样本设置和多种骨干模型下,GalLoP在11个数据集上的精度优于现有提示学习方法;此外,其在域泛化和OOD检测中均表现出强鲁棒性,甚至超过专用OOD检测方法。代码及复现说明:https://github.com/MarcLafon/gallop

英文摘要:

Prompt learning has been widely adopted to efficiently adapt vision-language models (VLMs), e.g. CLIP, for few-shot image classification. Despite their success, most prompt learning methods trade-off between classification accuracy and robustness, e.g. in domain generalization or out-of-distribution (OOD) detection. In this work, we introduce Global-Local Prompts (GalLoP), a new prompt learning method that learns multiple diverse prompts leveraging both global and local visual features. The training of the local prompts relies on local features with an enhanced vision-text alignment. To focus only on pertinent features, this local alignment is coupled with a sparsity strategy in the selection of the local features. We enforce diversity on the set of prompts using a new ``prompt dropout'' technique and a multiscale strategy on the local prompts. GalLoP outperforms previous prompt learning methods on accuracy on eleven datasets in different few shots settings and with various backbones. Furthermore, GalLoP shows strong robustness performances in both domain generalization and OOD detection, even outperforming dedicated OOD detection methods. Code and instructions to reproduce our results: https://github.com/MarcLafon/gallop.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑