Visual reprogramming (VR) reuses pre-trained vision models for downstream image classification tasks by adding trainable noise patterns to inputs. When applied to vision-language models (e.g., CLIP), existing VR approaches follow the same pipeline used in vision models (e.g., ResNet, ViT), where ground-truth class labels are inserted into fixed text templates to guide the optimization of VR patterns. This label-based approach, however, overlooks the rich information and diverse attribute-guided textual representations that CLIP can exploit, which may lead to the misclassification of samples. In this paper, we propose Attribute-based Visual Reprogramming (AttrVR) for CLIP, utilizing descriptive attributes (DesAttrs) and distinctive attributes (DistAttrs), which respectively represent common and unique feature descriptions for different classes. Besides, as images of the same class may reflect different attributes after VR, AttrVR iteratively refines patterns using the $k$-nearest DesAttrs and DistAttrs for each image sample, enabling more dynamic and sample-specific optimization. Theoretically, AttrVR is shown to reduce intra-class variance and increase inter-class separation. Empirically, it achieves superior performance in 12 downstream tasks for both ViT-based and ResNet-based CLIP. The success of AttrVR facilitates more effective integration of VR from unimodal vision models into vision-language models. Our code is available at https://github.com/tmlr-group/AttrVR.
BioMedVR: Confusion-Aware Mixture-of-Prompt Experts for Biomedical Visual Reprogramming
Guangdong Institute of Intelligence Science and Technology(广东智能科技研究院);Zhejiang University(浙江大学);Southeast University(东南大学);The Hong Kong Polytechnic University(香港理工大学);Institute of Science Tokyo(东京科学大学);Shanghai Jiao Tong University(上海交通大学)
CASA: Class-Agnostic Shared Attributes in Vision-Language Models for Efficient Incremental Object Detection
School of Electronic and Computer Engineering, Peking University(北京大学电子与计算机工程学院);Peng Cheng Laboratory(鹏城实验室);School of Computer Science, Peking University(北京大学计算机学院)
Position, Not Provenance: Separating Reasoning Mediation from Sycophancy in Medical Vision-Language Models
IEM Kolkata(印度工程管理学院 Kolkata校区);School of UEMK Kolkata(UEMK Kolkata学院);Heritage Institute of Technology Kolkata(加尔各答遗产技术学院);Indian Institute of Information Technology, Kalyani(卡利亚尼印度信息技术学院)
VLM-PAR: A Vision Language Model for Pedestrian Attribute Recognition
Department of Innovation Engineering(创新工程系);University of Salento, Italy(意大利萨伦托大学);Institute of Applied Sciences and Intelligent Systems - CNR(应用科学与智能系统研究所 - CNR);University of the Basque Country UPV/EHU(巴斯克国家大学UPV/EHU);IKERBASQUE, Basque Foundation for Science(伊基塔斯克巴塞克基金会);Sorbonne University Abu Dhabi(索邦大学阿布扎比分校)
Vision-to-Language Tasks Based on Attributes and Attention Mechanism
Northwestern Polytechnical University(西北工业大学);Xi’an Institute of Optics and Precision Mechanics, Chinese Academy of Sciences(中国科学院西安光学精密机械研究所);University of Chinese Academy of Sciences(中国科学院大学)
Toward Robust Medical Fairness: Debiased Dual-Modal Alignment via Text-Guided Attribute-Disentangled Prompt Learning for Vision-Language Models
National Engineering Laboratory for Integrated Aero-Space-Ground-Ocean Big Data Application Technology(集成空天地海大数据应用技术国家工程实验室);Northwestern Polytechnical University(西北工业大学);Huiying Medical Technology Company Ltd.(慧影医疗技术有限公司);The School of Computer Science(计算机学院);The University of Sydney(悉尼大学);Department of Computer Science and Engineering(计算机科学与工程系);The Chinese University of Hong Kong(香港中文大学)
The Eye of Sherlock Holmes: Uncovering User Private Attribute Profiling via Vision-Language Model Agentic Framework
Nanyang Technological University(南洋理工大学);Beijing University of Technology(北京理工大学);Hengxin Tech(恒心科技);Alibaba Group(阿里巴巴集团);Squirrel Ai Learning(squirrel AI 学习)
Pedestrian Attribute Recognition via CLIP based Prompt Vision-Language Fusion
School of Computer Science and Technology, Anhui University(安徽大学计算机科学与技术学院);Institute of Artificial Intelligence, Hefei Comprehensive National Science Center(合肥综合性国家科学中心人工智能研究院);Information Materials and Intelligent Sensing Laboratory of Anhui Province(安徽省信息材料与智能感知实验室);Anhui Provincial Key Laboratory of Multimodal Cognitive Computation(安徽省多模态认知计算重点实验室);School of Artificial Intelligence, Anhui University(安徽大学人工智能学院);Video Investigation Detachment of Hefei Public Security Bureau(合肥市公安局视频侦查支队)
SemPT: Semantic Prompt Tuning for Vision-Language Models
School of Computer Science and Artificial Intelligence, Wuhan Textile University(武汉纺织大学计算机科学与人工智能学院);School of Remote Sensing and Information Engineering, Wuhan University(武汉大学遥感与信息工程学院)
Diagnosing Dense Same-Class Attribute Misbinding in Large Vision-Language Models
Qilu University of Technology (Shandong Academy of Sciences)(齐鲁工业大学(山东省科学院));China Telecom Digital Intelligence Technology Co., Ltd.(中国电信数字智能科技有限公司);Shenyang Aerospace University(沈阳航空航天大学);University of Nottingham Ningbo China(宁波诺丁汉大学)
Causality-guided Prompt Learning for Vision-language Models via Visual Granulation
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences(多模态人工智能系统国家重点实验室,自动化研究所,中国科学院);School of Artificial Intelligence, University of Chinese Academy of Sciences(人工智能学院,中国科学院大学)
Social Perception of Faces in a Vision-Language Model
California Institute of Technology(加州理工学院);ETH Zurich(苏黎世联邦理工学院);Computational Social Science(计算社会科学);Swiss Data Science Center(瑞士数据科学中心);Empa(瑞士联邦材料科学与技术研究院)
Interpretable Zero-Shot Learning with Locally-Aligned Vision-Language Model
Mohamed bin Zayed University of AI(莫扎伊德大学人工智能学院);Huazhong University of Science and Technology(华中科技大学);Australian National University(澳大利亚国立大学);Linköping University(林雪平大学)
Unleashing the Capabilities of Large Vision-Language Models for Intelligent Perception of Roadside Infrastructure
State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing (LIESMARS)(信息工程测绘遥感国家重点实验室);Wuhan University(武汉大学);Hubei Luojia Laboratory(湖北珞珈实验室)
Dual Prompt Learning for Adapting Vision-Language Models to Downstream Image-Text Retrieval
College of Computer Science, Sichuan University(四川大学计算机学院);Engineering Research Center of Machine Learning and Industry Intelligence, Ministry of Education, Chengdu, China(教育部机器学习与产业智能工程研究中心);Deep NeuroCognition Lab, I2R and CFAR, Agency for Science, Technology and Research, Singapore(深度神经认知实验室,I2R和CFAR,科技研究局,新加坡)