Fine-tuning vision-language models (VLMs) like CLIP to downstream tasks is often necessary to optimize their performance. However, a major obstacle is the limited availability of labeled data. We study the use of pseudolabels, i.e., heuristic labels for unlabeled data, to enhance CLIP via prompt tuning. Conventional pseudolabeling trains a model on labeled data and then generates labels for unlabeled data. VLMs' zero-shot capabilities enable a "second generation" of pseudolabeling approaches that do not require task-specific training on labeled data. By using zero-shot pseudolabels as a source of supervision, we observe that learning paradigms such as semi-supervised, transductive zero-shot, and unsupervised learning can all be seen as optimizing the same loss function. This unified view enables the development of versatile training strategies that are applicable across learning paradigms. We investigate them on image classification tasks where CLIP exhibits limitations, by varying prompt modalities, e.g., textual or visual prompts, and learning paradigms. We find that (1) unexplored prompt tuning strategies that iteratively refine pseudolabels consistently improve CLIP accuracy, by 19.5 points in semi-supervised learning, by 28.4 points in transductive zero-shot learning, and by 15.2 points in unsupervised learning, and (2) unlike conventional semi-supervised pseudolabeling, which exacerbates model biases toward classes with higher-quality pseudolabels, prompt tuning leads to a more equitable distribution of per-class accuracy. The code to reproduce the experiments is at https://github.com/BatsResearch/menghini-neurips23-code.
Exploring Weak-to-Strong Generalization for CLIP-based Classification
School of Computing and Information Systems University of Melbourne, Australia（墨尔本大学计算机与信息系统学院）；School of Computing and Information Systems University of Melbournem, Australia（墨尔本大学计算机与信息系统学院）；School of Computer Science and Engineering Southeast University, China（东南大学计算机科学与工程学院）
DeltaEdit: Exploring Text-free Training for Text-Driven Image Manipulation
School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；CRIPAC, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所CRIPAC）；VIS, Baidu Inc.（百度公司VIS）
Google Research（谷歌研究院）；Cornell University（康奈尔大学）；University of Copenhagen（哥本哈根大学）
This work presents a self-supervised learning framework named TeG to explore Temporal Granularity in learning video representations. In TeG, we sample a long clip from a video and a short clip that lies inside the long clip. We then extract their dense temporal embeddings. The training objective consists of two parts: a fine-grained temporal learning objective to maximize the similarity between corresponding temporal embeddings in the short clip and the long clip, and a persistent temporal learning objective to pull together global embeddings of the two clips. Our study reveals the impact of t…
Advancing Vision-based Human Action Recognition: Exploring Vision-Language CLIP Model for Generalisation in Domain-Independent Tasks
Central University of Haryana（哈里亚纳中央大学）；University of New South Wales（新南威尔士大学）；Østfold University College（东福尔大学学院）；Norwegian University of Science and Technology（挪威科技大学）
Measuring the perception of visual content is a long-standing problem in computer vision. Many mathematical models have been developed to evaluate the look or quality of an image. Despite the effectiveness of such tools in quantifying degradations such as noise and blurriness levels, such quantification is loosely coupled with human language. When it comes to more abstract perception about the feel of visual content, existing methods can only rely on supervised models that are explicitly trained with labeled data collected via laborious user study. In this paper, we go beyond the conventional …
Recent learning-based video quality assessment (VQA) algorithms are expensive to implement due to the cost of data collection of human quality opinions, and are less robust across various scenarios due to the biases of these opinions. This motivates our exploration on opinion-unaware (a.k.a zero-shot) VQA approaches. Existing approaches only considers low-level naturalness in spatial or temporal domain, without considering impacts from high-level semantics. In this work, we introduce an explicit semantic affinity index for opinion-unaware VQA using text-prompts in the contrastive language-imag…
CLIP-PING: Boosting Lightweight Vision-Language Models with Proximus Intrinsic Neighbors Guidance
Kyung Hee University（庆熙大学）；The University of Danang—Vietnam-Korea University of Information and Communication Technology（岘港大学—越韩信息与通信技术大学）；Digital Science and Technology Institute（数字科学技术研究院）
Unveiling Backbone Effects in CLIP: Exploring Representational Synergies and Variances
Australian Institute for Machine Learning（澳大利亚机器学习研究所）；University of Adelaide（阿德莱德大学）；The University of Tokyo（东京大学）；AIST（日本产业技术综合研究所）；University of Chile（智利大学）；CENIA（智利国家人工智能研究中心）；IMFD（千年基础数据研究所）；Amazon（亚马逊）
Exploring Structured Semantic Prior for Multi Label Recognition with Incomplete Labels
Xidian University（西安电子科技大学）；Hangzhou Zhuoxi Institute of Brain and Intelligence（杭州卓兮脑与智能研究院）；Tsinghua University（清华大学）；BNRist（北京国家信息科学与技术研究中心）；the University of Sheffield（谢菲尔德大学）
Exploring CLIP's Dense Knowledge for Weakly Supervised Semantic Segmentation
Academy for Engineering and Technology, Fudan University（复旦大学工程技术研究院）；Digital Medical Research Center, School of Basic Medical Sciences, Fudan University（复旦大学基础医学院数字医学中心）；Shandong Computer Science Center (National Supercomputer Center in Jinan)（山东省计算机科学中心（国家济南超级计算中心））
The Hong Kong University of Science and Technology（香港科技大学）；University of Technology Sydney（悉尼科技大学）
Contrastive Language-Image Pre-training (CLIP) learns rich representations via readily available supervision of natural language. It improves the performance of downstream vision tasks, including but not limited to the zero-shot, long tail, segmentation, retrieval, caption, and video. However, the visual explainability of CLIP is rarely studied, especially for the raw feature map. To provide visual explanations of its predictions, we propose the Image-Text Similarity Map (ITSM). Based on it, we surprisingly find that CLIP prefers the background regions than the foregrounds, and shows erroneous…
Exploring the Potential of Contrastive Language-Image Pre-training for Multi-Source Remote Sensing Data
School of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；Space-based Computing System Research Center, Zhejiang Lab（之江实验室天基计算系统研究中心）
Self-adaptive vision-language model for 3D segmentation of pulmonary artery and vein
National Clinical Research Center for Cancer/Cancer Hospital Shenzhen Hospital（国家癌症临床研究中心/肿瘤医院深圳医院）；Key Laboratory of System Software (Chinese Academy of Sciences)（系统软件重点实验室（中国科学院））；State Key Laboratory of Computer Science（计算机科学国家重点实验室）；Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所）；Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences（中国科学院大学杭州高等研究院）；School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机科学与工程学院）；Guangzhou Jiayi Software Technology Co., Ltd.（广州嘉益软件科技有限公司）；R&D Center, Guangxi Huayi Artificial Intelligence Medical Technology Co., Ltd（广西华奕人工智能医疗科技有限公司研发中心）；Department of Computer Science, University of Exeter（埃克塞特大学计算机科学系）；Department of Cardiovascular & Metabolic Medicine, University of Liverpool（利物浦大学心血管与代谢医学系）
We investigate the efficacy of visual prompting to adapt large-scale models in vision. Following the recent approach from prompt tuning and adversarial reprogramming, we learn a single image perturbation such that a frozen model prompted with this perturbation performs a new task. Through comprehensive experiments, we demonstrate that visual prompting is particularly effective for CLIP and robust to distribution shift, achieving performance competitive with standard linear probes. We further analyze properties of the downstream dataset, prompt design, and output transformation in regard to ada…
C2P-CLIP: Injecting Category Common Prompt in CLIP to Enhance Generalization in Deepfake Detection
Institute of Information Science, Beijing Jiaotong University（北京交通大学信息科学研究所）；School of Information Science and Engineering, Yanshan University（燕山大学信息科学与工程学院）；School of Data Science, The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen)（香港中文大学（深圳）数据科学学院）
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Despite achieving state-of-the-art zero-shot performance, existing vision-language models still fall short of few-shot transfer ability on domain-specific problems. Classical fine-tuning often fails to prevent highly expressive models from exploiting spurious correlations. Although model-agnostic meta-learning (MAML) presents as a natural alternative for few-shot transfer learning, the expensive computation due to implicit second-order optimization limits its use on large-scale vision-language models such as CLIP. While much literature has been devoted to exploring alternative optimization str…
Can Large Language Models Grasp Event Signals? Exploring Pure Zero-Shot Event-based Recognition
School of Computer and Artificial Intelligence（计算机与人工智能学院）；Beijing Technology and Business University（北京工商大学）；School of Computer Science（计算机科学学院）；The University of Sydney（悉尼大学）
DA2Diff: Exploring Degradation-aware Adaptive Diffusion Priors for All-in-One Weather Restoration
School of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics（南京航空航天大学计算机科学与技术学院）；Collaborative Innovation Center of Novel Software Technology and Industrialization（新型软件技术与产业化协同创新中心）；College of Computer Science and Technology, Anhui University of Technology（安徽工业大学计算机科学与技术学院）；Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）
Learning visual knowledge from massive weakly-labeled web videos has attracted growing research interests thanks to the large corpus of easily accessible video data on the Internet. However, for video action recognition, the action of interest might only exist in arbitrary clips of untrimmed web videos, resulting in high label noises in the temporal space. To address this issue, we introduce a new method for pre-training video action recognition models using queried web videos. Instead of trying to filter out, we propose to convert the potential noises in these queried videos to useful supervi…
