发表机构
Chiba University（千叶大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
该研究提出PromptSpLiCE方法分析视觉-语言模型提示学习后的概念分布变化，发现概念分布变化与准确率提升正相关，还推导了局部梯度表达式解释相关几何直觉。
AI 中文摘要
提示学习通过优化连续提示向量来适配CLIP等视觉-语言模型，但学习到的提示难以用自然语言解释。我们提出PromptSpLiCE，这是一种事后方法，将每个类条件文本嵌入表示为固定自然语言词典中概念的稀疏组合。在提示学习前后使用同一词典，可对比它们的概念分布变化。我们在11个图像分类数据集上，针对代表性提示学习方法CoOp评估PromptSpLiCE。概念分布发生显著变化：平均而言，初始前10个概念中仅1.6个在学习后仍保留在前10。在各数据集上，分布变化与准确率提升呈正相关。我们还推导了局部梯度表达式，为与当前提示不同的图像对齐概念方向为何具有更高损失敏感性提供几何直观解释。
英文摘要
Prompt learning adapts vision-language models such as CLIP by optimizing continuous prompt vectors, but the learned prompts are difficult to interpret in natural language. We present PromptSpLiCE, a post-hoc method that expresses each class-conditioned text embedding as a sparse combination of concepts from a fixed natural-language dictionary. Using the same dictionary before and after prompt learning allows us to compare changes in their concept profiles. We evaluate PromptSpLiCE on CoOp, a representative prompt-learning method, across 11 image-classification datasets. The concept profiles change substantially: on average, only 1.6 of the initial top-10 concepts remain in the top 10 after learning. Across datasets, profile change is positively associated with accuracy gain. We also derive a local gradient expression that provides geometric intuition for why image-aligned concept directions distinct from the current prompt can have greater loss sensitivity.
Comments11 pages, 10 figures
