arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.13683cs.CVcs.AI
收藏

IntCoOp: 面向可解释性的视觉-语言提示调优

IntCoOp: Interpretability-Aware Vision-Language Prompt Tuning

University of Maryland, College Park(马里兰大学帕克分校)

Soumya Suvra Ghosal, Samyadeep Basu, Soheil Feizi, Dinesh Manocha

AI总结:

针对现有视觉-语言提示调优框架可解释性不足的问题,提出可解释方法IntCoOp,通过联合对齐属性级归纳偏置与类别嵌入提升性能,在10个数据集的16样本设置下较CoOp平均提升7.35%。

摘要:

CLIP等图像-文本对比模型可学习可迁移的鲁棒表征,用于零样本迁移至多种下游任务。但要获得优异的下游性能,需精心设计提示词,这是一项繁琐的工程工作。为解决手动提示工程的问题,研究者采用提示调优方法,即利用训练数据信息学习一组上下文向量。现有提示调优框架虽有效,但往往缺乏可解释性,限制了对图像组合本质的理解能力。本研究首先发现,在手动提示设计中引入组合属性(如“绿色的”树蛙)可显著提升图像-文本对齐分数。基于这一观察,我们提出一种新颖的可解释提示调优方法IntCoOp,该方法在提示调优过程中学习联合对齐属性级归纳偏置与类别嵌入。为评估方法的有效性,我们在少样本学习设置下,针对两个代表性任务评估IntCoOp:新类别泛化、未见过的域偏移。通过在CLIP上的10个下游数据集开展大量实验,我们发现引入属性级归纳偏置可带来优于现有前沿提示调优框架的性能。值得注意的是,在16样本设置下,IntCoOp在10个不同数据集上的平均性能较CoOp提升了7.35%。

英文摘要:

Image-text contrastive models such as CLIP learn transferable and robust representations for zero-shot transfer to a variety of downstream tasks. However, to obtain strong downstream performances, prompts need to be carefully curated, which can be a tedious engineering task. To address the issue of manual prompt engineering, prompt-tuning is used where a set of contextual vectors are learned by leveraging information from the training data. Despite their effectiveness, existing prompt-tuning frameworks often lack interpretability, thus limiting their ability to understand the compositional nature of images. In this work, we first identify that incorporating compositional attributes (e.g., a "green" tree frog) in the design of manual prompts can significantly enhance image-text alignment scores. Building upon this observation, we propose a novel and interpretable prompt-tuning method named IntCoOp, which learns to jointly align attribute-level inductive biases and class embeddings during prompt-tuning. To assess the effectiveness of our approach, we evaluate IntCoOp across two representative tasks in a few-shot learning setup: generalization to novel classes, and unseen domain shifts. Through extensive experiments across 10 downstream datasets on CLIP, we find that introducing attribute-level inductive biases leads to superior performance against state-of-the-art prompt tuning frameworks. Notably, in a 16-shot setup, IntCoOp improves CoOp by 7.35% in average performance across 10 diverse datasets.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑