The data appetite for Vision-Language Models (VLMs) has continuously scaled up from the early millions to billions today, which faces an untenable trade-off with data quality and inevitably introduces Noisy Correspondence (NC) samples. Undoubtedly, such semantically unrelated data significantly impairs the performance of VLMs. Previous efforts mainly address this challenge by estimating refined alignment for more precise guidance. However, such resource-intensive pipelines that train VLMs from scratch struggle to meet realistic data demands. In this paper, we present a brand new perspective that seeks to directly eliminate the harmful effects of NC in pre-trained VLMs. Specifically, we propose NCU, a Noisy Correspondence Unlearning fine-tuning framework that efficiently enhances VLMs' robustness by forgetting learned noisy knowledge. The key to NCU is learning the hardest negative information, which can provide explicit unlearning direction for both false positives and false negatives. Such twin goals unlearning process can be formalized into one unified optimal transport objective for fast fine-tuning. We validate our approach with the prevailing CLIP model over various downstream tasks. Remarkably, NCU surpasses the robust pre-trained method on zero-shot transfer while with lower computational overhead. The code will be released upon acceptance.
CLIP-AUTT: Test-Time Personalization with Action Unit Prompting for Fine-Grained Video Emotion Recognition
LIVIA, ILLS, Dept. of Systems Engineering, ETS Montreal（LIVIA, ILLS, 系统工程学院, 蒙特利尔高等技术学院）；LIVIA, Dept. of Software and IT Engineering, ETS Montreal（LIVIA, 软件与IT工程学院, 蒙特利尔高等技术学院）；Dept. of Computer Science, École Polytechnique（计算机科学系, 巴黎综合理工学院）
On the dynamic evolution of CLIP texture-shape bias and its relationship to human alignment and model robustness
Image Processing Lab, Universitat de Valencia, Spain（瓦伦西亚大学图像处理实验室）；Centro de Biomateriales e Ingenieria Tisular, Universitat Politecnica de Valencia, Spain（瓦伦西亚理工大学生物材料与组织工程中心）；Computer Vision Center, Spain（西班牙计算机视觉中心）；Universitat Autònoma de Barcelona, Spain（巴塞罗那自治大学）
Enhancing Vision-Language Model with Unmasked Token Alignment
Institute for AI Industry Research (AIR)（人工智能产业研究院）；Tsinghua University（清华大学）；Sensetime Research（商汤科技研究院）；CPII under InnoHK（InnoHK下属CPII）；Shanghai AI Laboratory（上海人工智能实验室）
XAI-CLIP: ROI-Guided Perturbation Framework for Explainable Medical Image Segmentation in Multimodal Vision-Language Models
King Fahd University of Petroleum and Minerals（国王法赫德石油和矿物大学）；Massachusetts Institute of Technology（麻省理工学院）；Imperial College London（伦敦帝国学院）；KFUPM-SDAIA Joint Research Centre for Artificial Intelligence（KFUPM-SDAIA联合人工智能研究中心）
Bridging Weakly-Supervised Learning and VLM Distillation: Noisy Partial Label Learning for Efficient Downstream Adaptation
Institute of Perceptual Intelligence, Peng Cheng Laboratory（感知智能研究院，鹏城实验室）；Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院，清华大学）
NLPrompt: Noise-Label Prompt Learning for Vision-Language Models
ShanghaiTech University（上海科技大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；RIKEN Center for Advanced Intelligence Project（日本RIKEN高级智能项目中心）；University of Technology Sydney（悉尼科技大学）；University of Melbourne（墨尔本大学）
Inspired by the remarkable zero-shot generalization capacity of vision-language pre-trained model, we seek to leverage the supervision from CLIP model to alleviate the burden of data labeling. However, such supervision inevitably contains the label noise, which significantly degrades the discriminative power of the classification model. In this work, we propose Transductive CLIP, a novel framework for learning a classification network with noisy labels from scratch. Firstly, a class-conditional contrastive learning mechanism is proposed to mitigate the reliance on pseudo labels and boost the t…
Direct Unlearning Optimization for Robust and Safe Text-to-Image Models
Seoul National University（首尔大学）；Korea University（高丽大学）；NAVER AI Lab（NAVER AI实验室）；NAVER Cloud（NAVER云）；Korea Institute for Advanced Study (KIAS)（韩国高等研究院）；AI Institute of Seoul National University or SNU AIIS（首尔大学AI研究所（SNU AIIS））
We present CLIPPER (Consistent LInking, Pruning, and Pairwise Error Rectification), a framework for robust data association in the presence of noise and outliers. We formulate the problem in a graph-theoretic framework using the notion of geometric consistency. State-of-the-art techniques that use this framework utilize either combinatorial optimization techniques that do not scale well to large-sized problems, or use heuristic approximations that yield low accuracy in high-noise, high-outlier regimes. In contrast, CLIPPER uses a relaxation of the combinatorial problem and returns solutions th…
A CLIP-Powered Framework for Robust and Generalizable Data Selection
National Key Laboratory for Novel Software Technology, Nanjing University（南京大学新型软件技术国家重点实验室）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；The Chinese University of Hong Kong（香港中文大学）
MIT CSAIL（麻省理工学院计算机科学与人工智能实验室）；Amazon Prime Video（亚马逊Prime Video）
The learning objective of vision-language approach of CLIP does not effectively account for the noisy many-to-many correspondences found in web-harvested image captioning datasets, which contributes to its compute and data inefficiency. To address this challenge, we introduce a novel training framework based on cross-modal contrastive learning that uses progressive self-distillation and soft image-text alignments to more efficiently learn robust representations from noisy data. Our model distills its own knowledge to dynamically generate soft-alignment targets for a subset of images and captio…
South China University of Technology（华南理工大学）；Pazhou Laboratory（琶洲实验室）；Baidu Inc.（百度公司）
We study self-supervised video representation learning, which is a challenging task due to 1) lack of labels for explicit supervision; 2) unstructured and noisy visual information. Existing methods mainly use contrastive loss with video clips as the instances and learn visual representation by discriminating instances from each other, but they need a careful treatment of negative pairs by either relying on large batch sizes, memory banks, extra modalities or customized mining strategies, which inevitably includes noisy data. In this paper, we observe that the consistency between positive sampl…
Traditional computer vision models are trained to predict a fixed set of predefined categories. Recently, natural language has been shown to be a broader and richer source of supervision that provides finer descriptions to visual concepts than supervised "gold" labels. Previous works, such as CLIP, use a simple pretraining task of predicting the pairings between images and text captions. CLIP, however, is data hungry and requires more than 400M image text pairs for training. We propose a data-efficient contrastive distillation method that uses soft labels to learn from noisy image-text pairs. …
Obtaining high resolution images from low resolution data with clipped noise is algorithmically challenging due to the ill-posed nature of the problem. So far such problems have hardly been tackled, and the few existing approaches use simplistic regularisers. We show the usefulness of two adaptive regularisers based on anisotropic diffusion ideas: Apart from evaluating the classical edge-enhancing anisotropic diffusion regulariser, we introduce a novel non-local one with one-sided differences and superior performance. It is termed sector diffusion. We combine it with all six variants of the cl…
