Recent advances in vision-language models (VLMs) trained on web-scale image-text pairs have enabled impressive zero-shot transfer across a diverse range of visual tasks. However, comprehensive and independent evaluation beyond standard benchmarks is essential to understand their robustness, limitations, and real-world applicability. This paper presents a systematic evaluation framework for VLMs under natural adversarial scenarios for diverse downstream tasks, which has been overlooked in previous evaluation works. We evaluate a wide range of VLMs (CLIP, robust CLIP, BLIP2, and SigLIP2) on curated adversarial datasets (typographic attacks, ImageNet-A, and natural language-induced adversarial examples). We measure the natural adversarial performance of selected VLMs for zero-shot image classification, semantic segmentation, and visual question answering. Our analysis reveals that robust CLIP models can amplify natural adversarial vulnerabilities, and CLIP models significantly reduce performance for natural language-induced adversarial examples. Additionally, we provide interpretable analyses to identify failure modes. We hope our findings inspire future research in robust and fair multimodal pattern recognition.
Breaking Déjà Vu: Independent Auditing of Visual Place Recognition through Vision-Language Reasoning
School of Electronics and Computer Science, University of Southampton（南安普顿大学电子与计算机科学学院）；School of Electrical Engineering and Computer Science, Queensland University of Technology（昆士兰科技大学电气工程与计算机科学学院）；School of Computer Science and Electronic Engineering, University of Essex（埃塞克斯大学计算机科学与电子工程学院）
RAPTOR+: A Visually Grounded Vision-Language Framework to Improve Clinical Trust and Auditability in Automated Cancer Referral Processing
Birmingham City University（伯明翰城市大学）；Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)（穆罕默德·本·扎耶德人工智能大学）；University Hospitals Birmingham NHS Foundation Trust（伯明翰大学医院 NHS 基础信托）；NHS England（英格兰国家卫生服务体系）
Edu-MMBias: A Three-Tier Multimodal Benchmark for Auditing Social Bias in Vision-Language Models under Educational Contexts
School of Computer Science and Technology, East China Normal University（东华师范大学计算机科学与技术学院）；Shanghai Institute of Artificial Intelligence for Education, East China Normal University（东华师范大学教育人工智能研究所）；Faculty of Education, East China Normal University（东华师范大学教育学院）
Auditing Medical Vision-Language Models on Chest Radiographs: Estimating Reference Agreement Across Institutions
University College Dublin（都柏林大学学院）；School of Computer Science, University College Dublin（都柏林大学学院计算机科学学院）；The Third Affiliated Hospital of Southern Medical University（南方医科大学第三附属医院）；Dublin City University（都柏林城市大学）
VLMInferSlow: Evaluating the Efficiency Robustness of Large Vision-Language Models as a Service
The Hong Kong University of Science and Technology（香港科学与技术大学）；Tongji University（同济大学）；Beijing Jiaotong University（北京交通大学）；Huawei（华为）；Tsinghua University（清华大学）；University of Chinese Academy of Sciences（中国科学院大学）
Pohang University of Science and Technology（浦项科技大学）
Vision transformers (ViTs) have recently demonstrated state-of-the-art performance in a variety of vision tasks, replacing convolutional neural networks (CNNs). Meanwhile, since ViT has a different architecture than CNN, it may behave differently. To investigate the reliability of ViT, this paper studies the behavior and robustness of ViT. We compared the robustness of CNN and ViT by assuming various image corruptions that may appear in practical vision tasks. We confirmed that for most image transformations, ViT showed robustness comparable to CNN or more improved. However, for contrast enhan…
Indian Institute of Technology Kharagpur（印度理工学院卡拉格普尔分校）
Computer vision applications like automated face detection are used for a variety of purposes ranging from unlocking smart devices to tracking potential persons of interest for surveillance. Audits of these applications have revealed that they tend to be biased against minority groups which result in unfair and concerning societal and political outcomes. Despite multiple studies over time, these biases have not been mitigated completely and have in fact increased for certain tasks like age prediction. While such systems are audited over benchmark datasets, it becomes necessary to evaluate thei…
Hyper Adversarial Tuning for Boosting Adversarial Robustness of Pretrained Large Vision Models
Zhejiang University（浙江大学）；Shanghai Maritime University（上海海事大学）；Zhejiang Gongshang University（浙江工商大学）；Nanjing University of Information Science and Technology（南京信息工程大学）
Université Paris-Saclay（巴黎萨克雷大学）；CentraleSupélec（中央理工-高等电力学院）；Inria（法国国家信息与自动化研究所）；Granular AI
Lipschitz bounded neural networks are certifiably robust and have a good trade-off between clean and certified accuracy. Existing Lipschitz bounding methods train from scratch and are limited to moderately sized networks (< 6M parameters). They require a fair amount of hyper-parameter tuning and are computationally prohibitive for large networks like Vision Transformers (5M to 660M parameters). Obtaining certified robustness of transformers is not feasible due to the non-scalability and inflexibility of the current methods. This work presents CertViT, a two-step proximal-projection method to a…
Alibaba Group（阿里巴巴集团）；Swinburne University of Technology（斯威本科技大学）；EPFL（洛桑联邦理工学院）
Recent advances on Vision Transformer (ViT) and its improved variants have shown that self-attention-based networks surpass traditional Convolutional Neural Networks (CNNs) in most vision tasks. However, existing ViTs focus on the standard accuracy and computation cost, lacking the investigation of the intrinsic influence on model robustness and generalization. In this work, we conduct systematic evaluation on components of ViTs in terms of their impact on robustness to adversarial examples, common corruptions and distribution shifts. We find some components can be harmful to robustness. By us…
National University of Singapore（新加坡国立大学）；NVIDIA（英伟达）；The University of Hong Kong（香港大学）；ASU（亚利桑那州立大学）；Caltech（加州理工学院）；ByteDance（字节跳动）
Recent studies show that Vision Transformers(ViTs) exhibit strong robustness against various corruptions. Although this property is partly attributed to the self-attention mechanism, there is still a lack of systematic understanding. In this paper, we examine the role of self-attention in learning robust representations. Our study is motivated by the intriguing properties of the emerging visual grouping in Vision Transformers, which indicates that self-attention may promote robustness through improved mid-level representations. We further propose a family of fully attentional networks (FANs) t…
Text Over Image: Auditing Multimodal Robustness in Synthetic Medical Image Detection
University of Notre Dame（圣母大学）；IBM Research（IBM研究院）；Boston Children’s Hospital（波士顿儿童医院）；Harvard Medical School（哈佛医学院）；Optum AI, UnitedHealth Group（Optum AI, 联合健康集团）
It has been observed that visual classification models often rely mostly on the image background, neglecting the foreground, which hurts their robustness to distribution changes. To alleviate this shortcoming, we propose to monitor the model's relevancy signal and manipulate it such that the model is focused on the foreground object. This is done as a finetuning step, involving relatively few samples consisting of pairs of images and their associated foreground masks. Specifically, we encourage the model's relevancy map (i) to assign lower relevance to background regions, (ii) to consider as m…
Columbia University（哥伦比亚大学）；Google Research（谷歌研究院）
Vision Transformer (ViT) is emerging as the state-of-the-art architecture for image recognition. While recent studies suggest that ViTs are more robust than their convolutional counterparts, our experiments find that ViTs trained on ImageNet are overly reliant on local textures and fail to make adequate use of shape information. ViTs thus have difficulties generalizing to out-of-distribution, real-world data. To address this deficiency, we present a simple and effective architecture modification to ViT's input layer by adding discrete tokens produced by a vector-quantized encoder. Different fr…
