Invariance to a broad array of image corruptions, such as warping, noise, or color shifts, is an important aspect of building robust models in computer vision. Recently, several new data augmentations have been proposed that significantly improve performance on ImageNet-C, a benchmark of such corruptions. However, there is still a lack of basic understanding on the relationship between data augmentations and test-time corruptions. To this end, we develop a feature space for image transforms, and then use a new measure in this space between augmentations and corruptions called the Minimal Sample Distance to demonstrate a strong correlation between similarity and performance. We then investigate recent data augmentations and observe a significant degradation in corruption robustness when the test-time corruptions are sampled to be perceptually dissimilar from ImageNet-C in this feature space. Our results suggest that test error can be improved by training on perceptually similar augmentations, and data augmentations may not generalize well beyond the existing benchmark. We hope our results and tools will allow for more robust progress towards improving robustness to image corruptions. We provide code at https://github.com/facebookresearch/augmentation-corruption.
University of California, Berkeley（加州大学伯克利分校）；Oregon State University（俄勒冈州立大学）
In this paper we establish rigorous benchmarks for image classifier robustness. Our first benchmark, ImageNet-C, standardizes and expands the corruption robustness topic, while showing which classifiers are preferable in safety-critical applications. Then we propose a new dataset called ImageNet-P which enables researchers to benchmark a classifier's robustness to common perturbations. Unlike recent robustness research, this benchmark evaluates performance on common corruptions and perturbations not worst-case adversarial perturbations. We find that there are negligible changes in relative cor…
Synthetic corruptions gathered into a benchmark are frequently used to measure neural network robustness to distribution shifts. However, robustness to synthetic corruption benchmarks is not always predictive of robustness to distribution shifts encountered in real-world applications. In this paper, we propose a methodology to build synthetic corruption benchmarks that make robustness estimations more correlated with robustness to real-world distribution shifts. Using the overlapping criterion, we split synthetic corruptions into categories that help to better understand neural network robustn…
Neural Networks are sensitive to various corruptions that usually occur in real-world applications such as blurs, noises, low-lighting conditions, etc. To estimate the robustness of neural networks to these common corruptions, we generally use a group of modeled corruptions gathered into a benchmark. Unfortunately, no objective criterion exists to determine whether a benchmark is representative of a large diversity of independent corruptions. In this paper, we propose a metric called corruption overlapping score, which can be used to reveal flaws in corruption benchmarks. Two corruptions overl…
Nanyang Technological University（南洋理工大学）；City University of Hong Kong（香港城市大学）
The state-of-the-art deep neural networks are vulnerable to common corruptions (e.g., input data degradations, distortions, and disturbances caused by weather changes, system error, and processing). While much progress has been made in analyzing and improving the robustness of models in image understanding, the robustness in video understanding is largely unexplored. In this paper, we establish a corruption robustness benchmark, Mini Kinetics-C and Mini SSV2-C, which considers temporal corruptions beyond spatial corruptions in images. We make the first attempt to conduct an exhaustive study on…
University of California, Berkeley（加州大学伯克利分校）；Oregon State University（俄勒冈州立大学）
In this paper we establish rigorous benchmarks for image classifier robustness. Our first benchmark, ImageNet-C, standardizes and expands the corruption robustness topic, while showing which classifiers are preferable in safety-critical applications. Unlike recent robustness research, this benchmark evaluates performance on commonplace corruptions not worst-case adversarial corruptions. We find that there are negligible changes in relative corruption robustness from AlexNet to ResNet classifiers, and we discover ways to enhance corruption robustness. Then we propose a new dataset called Icons-…
University of Michigan, Ann Arbor（密歇根大学安娜堡分校）；Lawrence Livermore National Laboratory（劳伦斯利弗莫尔国家实验室）；NVIDIA（英伟达）；ASU（亚利桑那州立大学（ASU））
Deep neural networks on 3D point cloud data have been widely used in the real world, especially in safety-critical applications. However, their robustness against corruptions is less studied. In this paper, we present ModelNet40-C, the first comprehensive benchmark on 3D point cloud corruption robustness, consisting of 15 common and realistic corruptions. Our evaluation shows a significant gap between the performances on ModelNet40 and ModelNet40-C for state-of-the-art (SOTA) models. To reduce the gap, we propose a simple but effective method by combining PointCutMix-R and TENT after evaluatin…
Tianjin University of Technology（天津理工大学）；Kyushu University（九州大学）；Alibaba Group（阿里巴巴集团）；Nanyang Technological University（南洋理工大学）；University of Alberta（阿尔伯塔大学）；Alberta Machine Intelligence Institute（阿尔伯塔机器智能研究所）
Network architecture search (NAS), in particular the differentiable architecture search (DARTS) method, has shown a great power to learn excellent model architectures on the specific dataset of interest. In contrast to using a fixed dataset, in this work, we focus on a different but important scenario for NAS: how to refine a deployed network's model architecture to enhance its robustness with the guidance of a few collected and misclassified examples that are degraded by some real-world unknown corruptions having a specific pattern (e.g., noise, blur, etc.). To this end, we first conduct an e…
Assessing and Enhancing Robustness of Deep Learning Models with Corruption Emulation in Digital Pathology
College of Engineering, Peking University（北京大学工学院）；Third Hospital, Peking University Health Science Center（北京大学医学部第三医院）；Institute of Biomedical Engineering, Beijing Institute of Collaborative Innovation（北京协同创新研究院生物医学工程研究所）
We introduce the MNIST-C dataset, a comprehensive suite of 15 corruptions applied to the MNIST test set, for benchmarking out-of-distribution robustness in computer vision. Through several experiments and visualizations we demonstrate that our corruptions significantly degrade performance of state-of-the-art computer vision models while preserving the semantic content of the test images. In contrast to the popular notion of adversarial robustness, our model-agnostic corruptions do not seek worst-case performance but are instead designed to be broad and diverse, capturing multiple failure modes…
DispBench: Benchmarking Disparity Estimation to Synthetic Corruptions
Data and Web Science Group, University of Mannheim（曼海姆大学数据与网络科学小组）；Max-Planck-Institute for Informatics, Saarland Informatics Campus（马克斯·普朗克研究所信息学研究所萨尔兰信息学校区）
本文提出DispBench基准测试工具，系统评估视差估计方法在对抗攻击和2D Common Corruptions等合成损坏下的鲁棒性，并通过迄今最广泛的分析揭示准确率、可靠性与泛化能力之间的关键相关性。
University of Alberta（阿尔伯塔大学）；Alberta Machine Intelligence Institute（阿尔伯塔机器智能研究所）；Kyushu University（九州大学）；Nanyang Technological University（南洋理工大学）；New York University（纽约大学）
Object detection through LiDAR-based point cloud has recently been important in autonomous driving. Although achieving high accuracy on public benchmarks, the state-of-the-art detectors may still go wrong and cause a heavy loss due to the widespread corruptions in the real world like rain, snow, sensor noise, etc. Nevertheless, there is a lack of a large-scale dataset covering diverse scenes and realistic corruption types with different severities to develop practical and robust point cloud detectors, which is challenging due to the heavy collection costs. To alleviate the challenge and start …
Artificial Intelligence Research Institute, Shenzhen MSU-BIT University（深圳北理莫斯科大学人工智能研究院）；Guangdong-Hong Kong-Macao Joint Laboratory for Emotional Intelligence and Pervasive Computing（粤港澳情感智能与普适计算联合实验室）；Shenzhen University（深圳大学）；South China University of Technology（华南理工大学）
Temporal action detection (TAD) aims to locate action positions and recognize action categories in long-term untrimmed videos. Although many methods have achieved promising results, their robustness has not been thoroughly studied. In practice, we observe that temporal information in videos can be occasionally corrupted, such as missing or blurred frames. Interestingly, existing methods often incur a significant performance drop even if only one frame is affected. To formally evaluate the robustness, we establish two temporal corruption robustness benchmarks, namely THUMOS14-C and ActivityNet-…
NLPR, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所模式识别国家重点实验室）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Youtu Lab, Tencent Inc.（腾讯优图实验室）；CASIA-LLvision Joint Lab（中国科学院自动化研究所与联影智能联合实验室）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Shenzhen Research Institute of Big Data（深圳市大数据研究院）
Big progress has been achieved in domain adaptation in decades. Existing works are always based on an ideal assumption that testing target domain are i.i.d. with training target domains. However, due to unpredictable corruptions (e.g., noise and blur) in real data like web images, domain adaptation methods are increasingly required to be corruption robust on target domains. In this paper, we investigate a new task, Corruption-agnostic Robust Domain Adaptation (CRDA): to be accurate on original data and robust against unavailable-for-training corruptions on target domains. This task is non-triv…
Improving Mixup Calibration with Wasserstein Distributionally Robust Optimization
Department of Mathematics & Statistics（数学与统计学系）；Boston University（波士顿大学）；Department of Electrical and Computer Engineering, Division of Systems Engineering（电气与计算机工程系，系统工程 division）；Department of Biomedical Engineering（生物医学工程系）；Faculty of Computing & Data Sciences（计算与数据科学学院）
Data augmentation has been actively studied for robust neural networks. Most of the recent data augmentation methods focus on augmenting datasets during the training phase. At the testing phase, simple transformations are still widely used for test-time augmentation. This paper proposes a novel instance-level test-time augmentation that efficiently selects suitable transformations for a test input. Our proposed method involves an auxiliary module to predict the loss of each possible transformation given the input. Then, the transformations having lower predicted losses are applied to the input…
Robo3D: Towards Robust and Reliable 3D Perception against Corruptions
Shanghai AI Laboratory（上海人工智能实验室）；National University of Singapore（新加坡国立大学）；Hochschule Bremerhaven（不来梅哈芬应用技术大学）；East China Normal University（华东师范大学）；The University of Hong Kong（香港大学）；Nanyang Technological University（南洋理工大学）
Swiss Federal Institute of Technology (EPFL)（瑞士联邦理工学院（洛桑联邦理工学院））
We introduce a set of image transformations that can be used as corruptions to evaluate the robustness of models as well as data augmentation mechanisms for training neural networks. The primary distinction of the proposed transformations is that, unlike existing approaches such as Common Corruptions, the geometry of the scene is incorporated in the transformations -- thus leading to corruptions that are more likely to occur in the real world. We also introduce a set of semantic corruptions (e.g. natural object occlusions). We show these transformations are `efficient' (can be computed on-the-…
Nanyang Technological University（南洋理工大学）；New York University（纽约大学）；Tianjin University（天津大学）；Centre for Frontier AI Research (CFAR), A*STAR（新加坡科技研究局前沿人工智能研究中心（CFAR））；Institute of High Performance Computing (IHPC), A*STAR（新加坡科技研究局高性能计算研究所（IHPC））
DSRC: Learning Density-insensitive and Semantic-aware Collaborative Representation against Corruptions
Academy for Engineering and Technology, Fudan University（复旦大学工程技术研究院）；Innovation Platform for Academicians of Hainan Province（海南省院士创新平台）；Duke Kunshan University（昆山杜克大学）；Chongqing Changan Automobile Co., Ltd.（重庆长安汽车股份有限公司）
Modern neural networks excel at image classification, yet they remain vulnerable to common image corruptions such as blur, speckle noise or fog. Recent methods that focus on this problem, such as AugMix and DeepAugment, introduce defenses that operate in expectation over a distribution of image corruptions. In contrast, the literature on $\ell_p$-norm bounded perturbations focuses on defenses against worst-case corruptions. In this work, we reconcile both approaches by proposing AdversarialAugment, a technique which optimizes the parameters of image-to-image models to generate adversarially co…
Stylized Synthetic Augmentation further improves Corruption Robustness
Institute of Industrial Automation and Software Engineering, University of Stuttgart, Stuttgart, Germany（工业自动化与软件工程学院，斯图加特大学，斯图加特，德国）；Federal Institute for Occupational Safety and Health (BAuA)（联邦职业安全与健康研究所）
Diverse data augmentation strategies are a natural approach to improving robustness in computer vision models against unforeseen shifts in data distribution. However, the ability to tailor such strategies to inoculate a model against specific classes of corruptions or attacks -- without incurring substantial losses in robustness against other classes of corruptions -- remains elusive. In this work, we successfully harden a model against Fourier-based attacks, while producing superior-to-AugMix accuracy and calibration results on both the CIFAR-10-C and CIFAR-100-C datasets; classification erro…
Ecole Polytechnique Fédérale de Lausanne (EPFL)（洛桑联邦理工学院）；ETH Zürich（苏黎世联邦理工学院）；Imperial College London（伦敦帝国理工学院）
Despite their impressive performance on image classification tasks, deep networks have a hard time generalizing to unforeseen corruptions of their data. To fix this vulnerability, prior works have built complex data augmentation strategies, combining multiple methods to enrich the training data. However, introducing intricate design choices or heuristics makes it hard to understand which elements of these methods are indeed crucial for improving robustness. In this work, we take a step back and follow a principled approach to achieve robustness to common corruptions. We propose PRIME, a genera…
Nanyang Technological University（南洋理工大学）；Agency for Science, Technology and Research（新加坡科技研究局）
Model robustness is vital for the reliable deployment of machine learning models in real-world applications. Recent studies have shown that data augmentation can result in model over-relying on features in the low-frequency domain, sacrificing performance against low-frequency corruptions, highlighting a connection between frequency and robustness. Here, we take one step further to more directly study the frequency bias of a model through the lens of its Jacobians and its implication to model robustness. To achieve this, we propose Jacobian frequency regularization for models' Jacobians to hav…
Recently, vision transformers and MLP-based models have been developed in order to address some of the prevalent weaknesses in convolutional neural networks. Due to the novelty of transformers being used in this domain along with the self-attention mechanism, it remains unclear to what degree these architectures are robust to corruptions. Despite some works proposing that data augmentation remains essential for a model to be robust against corruptions, we propose to explore the impact that the architecture has on corruption robustness. We find that vision transformer architectures are inherent…
Fudan University（复旦大学）；Microsoft Research Asia（微软亚洲研究院）
With Vision Transformers (ViTs) making great advances in a variety of computer vision tasks, recent literature have proposed various variants of vanilla ViTs to achieve better efficiency and efficacy. However, it remains unclear how their unique architecture impact robustness towards common corruptions. In this paper, we make the first attempt to probe into the robustness gap among ViT variants and explore underlying designs that are essential for robustness. Through an extensive and rigorous benchmarking, we demonstrate that simple architecture designs such as overlapping patch embedding and …
VITA: A Multi-Source Vicinal Transfer Augmentation Method for Out-of-Distribution Generalization
Southern University of Science and Technology（南方科技大学）；The University of Sydney（悉尼大学）；JD Explore Academy, JD.com Inc（京东探索研究院）；National Center for Artificial Intelligence, Saudi Data and Artificial Intelligence Authority（沙特数据与人工智能局国家人工智能中心）
Achieving robustness to distributional shift is a longstanding and challenging goal of computer vision. Data augmentation is a commonly used approach for improving robustness, however robustness gains are typically not uniform across corruption types. Indeed increasing performance in the presence of random noise is often met with reduced performance on other corruptions such as contrast change. Understanding when and why these sorts of trade-offs occur is a crucial step towards mitigating them. Towards this end, we investigate recently observed trade-offs caused by Gaussian data augmentation a…
Exploring the Robustness of Human Parsers Towards Common Corruptions
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Sun Yat-sen University（中山大学）；Westlake University（西湖大学）；OPPO；Tsinghua University（清华大学）
The literature on robustness towards common corruptions shows no consensus on whether adversarial training can improve the performance in this setting. First, we show that, when used with an appropriately selected perturbation radius, $\ell_p$ adversarial training can serve as a strong baseline against common corruptions improving both accuracy and calibration. Then we explain why adversarial training performs better than data augmentation with simple Gaussian noise which has been observed to be a meaningful baseline on common corruptions. Related to this, we identify the $σ$-overfitting pheno…
Improving Robustness of LiDAR-Camera Fusion Model against Weather Corruption from Fusion Strategy Perspective
Nanyang Technological University（南洋理工大学）；East China Normal University（华东师范大学）；Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局）；New York University（纽约大学）
Image text retrieval is a task to search for the proper textual descriptions of the visual world and vice versa. One challenge of this task is the vulnerability to input image and text corruptions. Such corruptions are often unobserved during the training, and degrade the retrieval model decision quality substantially. In this paper, we propose a novel image text retrieval technique, referred to as robust visual semantic embedding (RVSE), which consists of novel image-based and text-based augmentation techniques called semantic preserving augmentation for image (SPAugI) and text (SPAugT). Sinc…
Convolutional Neural Network's (CNN's) performance disparity on clean and corrupted datasets has recently come under scrutiny. In this work, we analyse common corruptions in the frequency domain, i.e., High Frequency corruptions (HFc, e.g., noise) and Low Frequency corruptions (LFc, e.g., blur). Although a simple solution to HFc is low-pass filtering, ReLU -- a widely used Activation Function (AF), does not have any filtering mechanism. In this work, we instill low-pass filtering into the AF (LP-ReLU) to improve robustness against HFc. To deal with LFc, we complement LP-ReLU with Discrete Cosi…
