Contrastive Language-Image Pre-training (CLIP) models have shown significant potential, particularly in zero-shot classification across diverse distribution shifts. Building on existing evaluations of overall classification robustness, this work aims to provide a more comprehensive assessment of CLIP by introducing several new perspectives. First, we investigate their robustness to variations in specific visual factors. Second, we assess two critical safety objectives--confidence uncertainty and out-of-distribution detection--beyond mere classification accuracy. Third, we evaluate the finesse with which CLIP models bridge the image and text modalities. Fourth, we extend our examination to 3D awareness in CLIP models, moving beyond traditional 2D image understanding. Finally, we explore the interaction between vision and language encoders within modern large multimodal models (LMMs) that utilize CLIP as the visual backbone, focusing on how this interaction impacts classification robustness. In each aspect, we consider the impact of six factors on CLIP models: model architecture, training distribution, training set size, fine-tuning, contrastive loss, and test-time prompts. Our study uncovers several previously unknown insights into CLIP. For instance, the architecture of the visual encoder in CLIP plays a significant role in their robustness against 3D corruption. CLIP models tend to exhibit a bias towards shape when making predictions. Moreover, this bias tends to diminish after fine-tuning on ImageNet. Vision-language models like LLaVA, leveraging the CLIP vision encoder, could exhibit benefits in classification performance for challenging categories over CLIP alone. Our findings are poised to offer valuable guidance for enhancing the robustness and reliability of CLIP models.
Complementary Text-Guided Attention for Zero-Shot Adversarial Robustness
School of Computer Science and Engineering, Tianjin University of Technology（天津理工大学计算机科学与工程学院）；State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所多模态人工智能系统国家重点实验室）；School of Artificial Intelligence, University of the Chinese Academy of Sciences（中国科学院大学人工智能学院）
Investigating Adversarial Robustness of Multi-modal Large Language Models
Mohamed Bin Zayed University of AI, UAE（穆罕默德·本·扎耶德人工智能大学，阿联酋）；Khalifa University, UAE（哈利法大学，阿联酋）；Australian National University, Australia（澳大利亚国立大学，澳大利亚）
On the dynamic evolution of CLIP texture-shape bias and its relationship to human alignment and model robustness
Image Processing Lab, Universitat de Valencia, Spain（瓦伦西亚大学图像处理实验室）；Centro de Biomateriales e Ingenieria Tisular, Universitat Politecnica de Valencia, Spain（瓦伦西亚理工大学生物材料与组织工程中心）；Computer Vision Center, Spain（西班牙计算机视觉中心）；Universitat Autònoma de Barcelona, Spain（巴塞罗那自治大学）
A vision-language foundation model pretrained on very large-scale image-text paired data has the potential to provide generalizable knowledge representation for downstream visual recognition and detection tasks, especially on supplementing the undersampled categories in downstream model training. Recent studies utilizing CLIP for object detection have shown that a two-stage detector design typically outperforms a one-stage detector, while requiring more expensive training resources and longer inference time. In this work, we propose a one-stage detector GridCLIP that narrows its performance ga…
University of Washington（华盛顿大学）；Allen Institute for Artificial Intelligence（艾伦人工智能研究所）
Web-crawled datasets have enabled remarkable generalization capabilities in recent image-text models such as CLIP (Contrastive Language-Image pre-training) or Flamingo, but little is known about the dataset creation processes. In this work, we introduce a testbed of six publicly available data sources - YFCC, LAION, Conceptual Captions, WIT, RedCaps, Shutterstock - to investigate how pre-training distributions induce robustness in CLIP. We find that the performance of the pre-training data varies substantially across distribution shifts, with no single data source dominating. Moreover, we syst…
University of Washington（华盛顿大学）；Amazon（亚马逊）；Allen Institute for Artificial Intelligence（艾伦人工智能研究所）
Contrastively trained language-image models such as CLIP, ALIGN, and BASIC have demonstrated unprecedented robustness to multiple challenging natural distribution shifts. Since these language-image models differ from previous training approaches in several ways, an important question is what causes the large robustness gains. We answer this question via a systematic experimental investigation. Concretely, we study five different possible causes for the robustness gains: (i) the training set size, (ii) the training distribution, (iii) language supervision at training time, (iv) language supervi…
School of Biomedical Engineering, Southern Medical University（南方医科大学生物医学工程学院）；School of Biomedical Engineering, Shanghai Jiao Tong University（上海交通大学生物医学工程学院）；Shanghai United Imaging Intelligence Co., Ltd.（上海联影智能医疗科技有限公司）；Shanghai Clinical Research and Trial Center（上海临床研究与试验中心）
Proxy Robustness in Vision Language Models is Effortlessly Transferable
Chongqing Key Laboratory of Bio-perception（重庆生物感知实验室）；Multimodal Intelligent Information Processing, Chongqing University, Chongqing 401331, China（多模态智能信息处理，重庆大学，重庆401331，中国）；School of Microelectronics（微电子学院）；Communication Engineering, Chongqing University, Chongqing 401331, China（通信工程，重庆大学，重庆401331，中国）；School of Data Science, Lingnan University, Hong Kong（数据科学学院，岭南大学，香港）
A Sober Look at the Robustness of CLIPs to Spurious Features
Hong Kong Baptist University（香港浸会大学）；The Hong Kong University of Science and Technology（香港科技大学）；The Chinese University of Hong Kong（香港中文大学）；University of Washington（华盛顿大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
AGC: Adaptive Geodesic Correction for Adversarial Robustness on Vision-Language Models
NLPR & MAIS, Institute of Automation, Chinese Academy of Sciences（自动化研究所国家工程研究中心与人工智能院，中国科学院）；School of Computer Science and Engineering, Central South University（中南大学计算机科学与工程学院）；University of Chinese Academy of Sciences（中国科学院大学）
Alibaba Group（阿里巴巴集团）；Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；Zhejiang University（浙江大学）
Contrastive Language-Image Pre-trained (CLIP) models have zero-shot ability of classifying an image belonging to "[CLASS]" by using similarity between the image and the prompt sentence "a [CONTEXT] of [CLASS]". Based on exhaustive text cues in "[CONTEXT]", CLIP model is aware of different contexts, e.g. background, style, viewpoint, and exhibits unprecedented robustness against a wide range of distribution shifts. However, recent works find further fine-tuning of CLIP models improves accuracy but sacrifices the robustness on downstream tasks. We conduct an empirical investigation to show fine-…
Bridging the Task Gap: Multi-Task Adversarial Transferability in CLIP and Its Derivatives
Shenzhen Campus of Sun Yat-sen University, China（中山大学深圳校区）；The National University of Singapore, Singapore（新加坡国立大学）；National Key Laboratory of Science and Technology on Information System Security, China（信息系统安全科学技术国家重点实验室）
Self-Calibrated Consistency can Fight Back for Adversarial Robustness in Vision-Language Models
Guangdong Institute of Intelligence Science and Technology（广东智能科学与技术研究院）；Agency for Science, Technology and Research（科技研究局）；School of Information Technology（信息技术学院）
CLIP (Contrastive Language-Image Pre-training) models developed by OpenAI have achieved outstanding results on various image recognition and retrieval tasks, displaying strong zero-shot performance. This means that they are able to perform effectively on tasks for which they have not been explicitly trained. Inspired by the success of OpenAI CLIP, a new publicly available dataset called LAION-5B was collected which resulted in the development of open ViT-H/14, ViT-G/14 models that outperform the OpenAI L/14 model. The LAION-5B dataset also released an approximate nearest neighbor index, with a…
