Image-text contrastive models such as CLIP learn transferable and robust representations for zero-shot transfer to a variety of downstream tasks. However, to obtain strong downstream performances, prompts need to be carefully curated, which can be a tedious engineering task. To address the issue of manual prompt engineering, prompt-tuning is used where a set of contextual vectors are learned by leveraging information from the training data. Despite their effectiveness, existing prompt-tuning frameworks often lack interpretability, thus limiting their ability to understand the compositional nature of images. In this work, we first identify that incorporating compositional attributes (e.g., a "green" tree frog) in the design of manual prompts can significantly enhance image-text alignment scores. Building upon this observation, we propose a novel and interpretable prompt-tuning method named IntCoOp, which learns to jointly align attribute-level inductive biases and class embeddings during prompt-tuning. To assess the effectiveness of our approach, we evaluate IntCoOp across two representative tasks in a few-shot learning setup: generalization to novel classes, and unseen domain shifts. Through extensive experiments across 10 downstream datasets on CLIP, we find that introducing attribute-level inductive biases leads to superior performance against state-of-the-art prompt tuning frameworks. Notably, in a 16-shot setup, IntCoOp improves CoOp by 7.35% in average performance across 10 diverse datasets.
Exploring Interpretability for Visual Prompt Tuning with Cross-layer Concepts
School of Computer Science and Technology, Tongji University(同济大学计算机科学与技术学院);Microsoft Research Asia(微软亚洲研究院);School of Telecommunication and Engineering, Xidian University(西安电子科技大学电信与工程学院)
Disentangling Hallucinations: Orthogonal Semantic Projection for Robust Interpretability
U2IS, ENSTA, Institut Polytechnique de Paris(巴黎综合理工学院ENSTA学院U2IS实验室);ISIR, Université Sorbonne, Pierre et Marie Curie(索邦大学皮埃尔和玛丽·居里分校ISIR实验室);AMIAD, Pôle Recherche(AMIAD研究部)
Interpretable Few-Shot Retinal Disease Diagnosis with Concept-Guided Prompting of Vision-Language Models
Monash University(莫纳什大学);Centre for Eye Research Australia(澳大利亚眼科研究中心);Royal Victorian Eye and Ear Hospital(皇家维多利亚眼耳医院);The University of Melbourne(墨尔本大学);The Hong Kong Polytechnic University(香港理工大学);Centre for Eye and Vision Research(眼与视觉研究中心);Indian Institute of Technology Bombay(印度理工学院孟买分校);Airdoc-Monash Research Lab(Airdoc-莫纳什研究实验室)
LLM-Powered Flood Depth Estimation from Social Media Imagery: A Vision-Language Model Framework with Mechanistic Interpretability for Transportation Resilience
Prompt Your Brain: Scaffold Prompt Tuning for Efficient Adaptation of fMRI Pre-trained Model
Yong Loo Lin School of Medicine, National University of Singapore(新加坡国立大学杨潞龄医学院);Department of Medicine, Yong Loo Lin School of Medicine, National University of Singapore(新加坡国立大学杨潞龄医学院医学系);NUS Graduate School, National University of Singapore(新加坡国立大学研究生院)
EDITOR: Effective and Interpretable Prompt Inversion for Text-to-Image Diffusion Models
University of Massachusetts, Amherst(马萨诸塞大学阿姆赫斯特分校);Georgia Institute of Technology(佐治亚理工学院);Rutgers University(罗格斯大学);University of Utah(犹他大学);Dolby Laboratories(杜比实验室)
Attribute-formed Class-specific Concept Space: Endowing Language Bottleneck Model with Better Interpretability and Scalability
University of Electronic Science and Technology of China(电子科技大学);Southwest Jiaotong University(西南交通大学);University of Minnesota(明尼苏达大学);Harvard University(哈佛大学);Nanyang Technological University(南洋理工大学)