arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.08347cs.CV
收藏

通过稀疏最优传输实现局部-全局提示学习

Local-Global Prompt Learning via Sparse Optimal Transport

Deniz Kizaroğlu, Ülku Tuncer Küçüktas, Emre Çakmakyurdu, Alptekin Temizel

AI总结:

SOT-GLP通过稀疏最优传输实现局部-全局提示学习,提升少样本分类和分布外检测性能。

摘要:

视觉语言模型(VLMs)如CLIP的少样本适应通常依赖于学习与全局图像嵌入匹配的文本提示。最近的研究扩展了这一范式,通过引入局部图像-文本对齐来捕捉细粒度的视觉线索,但这些方法往往为每个提示独立选择局部区域,导致冗余的局部特征使用和提示重叠。我们提出了SOT-GLP,引入了共享的稀疏补丁支持和平衡的最优传输分配,以显式地将显著的视觉区域分配给类特定的局部提示,同时保持全局对齐。我们的方法学习共享的全局提示和类特定的局部提示。全局分支维持标准的图像-文本匹配以实现稳健的类别级对齐。局部分支使用V-V注意力构建类条件的稀疏补丁集,并通过平衡的熵最优传输将其对齐到多个类特定的提示,从而得到一个软分割的补丁集,防止提示重叠和崩溃。我们在两个互补的目标上评估了我们的方法:(i)在11个标准基准上的少样本分类精度,以及(ii)分布外(OOD)检测。在标准11数据集基准上使用16-shot ViT-B/16,SOT-GLP实现了85.1%的平均准确率,优于先前的提示学习方法。我们发现提示学习中存在一个明显的准确率-鲁棒性权衡:虽然可学习的投影在分布内优化拟合,但它们会改变基础特征空间。我们证明,无投影的局部对齐保留了CLIP流形的本原几何,从而获得了最先进的OOD检测性能(94.2% AUC),超过了完全适应的模型。实现可在:https://github.com/Deniz2304988/SOT-GLP上找到。

英文摘要:

Few-shot adaptation of vision-language models (VLMs) like CLIP typically relies on learning textual prompts matched to global image embeddings. Recent works extend this paradigm by incorporating local image-text alignment to capture fine-grained visual cues, yet these approaches often select local regions independently for each prompt, leading to redundant local feature usage and prompt overlap. We propose SOT-GLP, which introduces a shared sparse patch support and balanced optimal transport allocation to explicitly partition salient visual regions among class-specific local prompts while preserving global alignment. Our method learns shared global prompts and class-specific local prompts. The global branch maintains standard image-text matching for robust category-level alignment. The local branch constructs a class-conditioned sparse patch set using V-V attention and aligns it to multiple class-specific prompts via balanced entropic optimal transport, yielding a soft partition of patches that prevents prompt overlap and collapse. We evaluate our method on two complementary objectives: (i) few-shot classification accuracy on 11 standard benchmarks and (ii) out-of-distribution (OOD) detection. On the standard 11-dataset benchmark with 16-shot ViT-B/16, SOT-GLP achieves 85.1% average accuracy, outperforming prior prompt-learning methods. We identify a distinct accuracy-robustness trade-off in prompt learning: while learnable projections optimize in-distribution fit, they alter the foundational feature space. We demonstrate that a projection-free local alignment preserves the native geometry of the CLIP manifold, yielding state-of-the-art OOD detection performance (94.2% AUC) that surpasses fully adapted models. Implementation available at: https://github.com/Deniz2304988/SOT-GLP

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑