Counterfactual Reasoning for Multi-Label Image Classification via Patching-Based Training
Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；RIKEN Center for Advanced Intelligence Project（理化学研究所高级智能项目中心）；The University of Tokyo（东京大学）
The key to multi-label image classification (MLC) is to improve model performance by leveraging label correlations. Unfortunately, it has been shown that overemphasizing co-occurrence relationships can cause the overfitting issue of the model, ultimately leading to performance degradation. In this paper, we provide a causal inference framework to show that the correlative features caused by the target object and its co-occurring objects can be regarded as a mediator, which has both positive and negative impacts on model predictions. On the positive side, the mediator enhances the recognition performance of the model by capturing co-occurrence relationships; on the negative side, it has the harmful causal effect that causes the model to make an incorrect prediction for the target object, even when only co-occurring objects are present in an image. To address this problem, we propose a counterfactual reasoning method to measure the total direct effect, achieved by enhancing the direct effect caused only by the target object. Due to the unknown location of the target object, we propose patching-based training and inference to accomplish this goal, which divides an image into multiple patches and identifies the pivot patch that contains the target object. Experimental results on multiple benchmark datasets with diverse configurations validate that the proposed method can achieve state-of-the-art performance.
Graduate School of System Informatics, Kobe University（神户大学系统信息学研究生院）
Deep convolutional neural networks (CNNs) have achieved remarkable results in image processing tasks. However, their high expression ability risks overfitting. Consequently, data augmentation techniques have been proposed to prevent overfitting while enriching datasets. Recent CNN architectures with more parameters are rendering traditional data augmentation techniques insufficient. In this study, we propose a new data augmentation technique called random image cropping and patching (RICAP) which randomly crops four images and patches them to create a new training image. Moreover, RICAP mixes …
Individual treatment effect (ITE) represents the expected improvement in the outcome of taking a particular action to a particular target, and plays important roles in decision making in various domains. However, its estimation problem is difficult because intervention studies to collect information regarding the applied treatments (i.e., actions) and their outcomes are often quite expensive in terms of time and monetary costs. In this study, we consider a semi-supervised ITE estimation problem that exploits more easily-available unlabeled instances to improve the performance of ITE estimation…
There exists an apparent negative correlation between performance and interpretability of deep learning models. In an effort to reduce this negative correlation, we propose a Born Identity Network (BIN), which is a post-hoc approach for producing multi-way counterfactual maps. A counterfactual map transforms an input sample to be conditioned and classified as a target label, which is similar to how humans process knowledge through counterfactual thinking. For example, a counterfactual map can localize hypothetical abnormalities from a normal brain image that may cause it to be diagnosed with a…
Australian Institute for Machine Learning（澳大利亚机器学习研究所）；University of Adelaide（阿德莱德大学）
One of the primary challenges limiting the applicability of deep learning is its susceptibility to learning spurious correlations rather than the underlying mechanisms of the task of interest. The resulting failure to generalise cannot be addressed by simply using more data from the same distribution. We propose an auxiliary training objective that improves the generalization capabilities of neural networks by leveraging an overlooked supervisory signal found in existing datasets. We use pairs of minimally-different examples with different labels, a.k.a counterfactual or contrasting examples, …
Attri-Net: A Globally and Locally Inherently Interpretable Model for Multi-Label Classification Using Class-Specific Counterfactuals
Cluster of Excellence: Machine Learning - New Perspectives for Science, University of Tübingen（机器学习新视角科学卓越中心，图宾根大学）；Department of Computer Science, Friedrich-Alexander-Universität Erlangen-Nürnberg（计算机科学系，埃尔兰根-纽伦堡弗里德里希-亚历山大大学）；Hertie Institute for Artificial Intelligence in Brain Health, University of Tübingen（脑健康人工智能研究所，图宾根大学）；Department of Diabetes, Endocrinology, Nutritional Medicine and Metabolism, Inselspital, Bern University Hospital, University of Bern（糖尿病、内分泌学、营养医学和代谢学系，伯恩大学医院，伯恩大学）；Faculty of Health Sciences and Medicine, University of Lucerne（健康科学与医学学院，卢塞恩大学）
When an image classifier outputs a wrong class label, it can be helpful to see what changes in the image would lead to a correct classification. This is the aim of algorithms generating counterfactual explanations. However, there is no easily scalable method to generate such counterfactuals. We develop a new algorithm providing counterfactual explanations for large image classifiers trained with spectral normalisation at low computational cost. We empirically compare this algorithm against baselines from the literature; our novel algorithm consistently finds counterfactuals that are much close…
University of Toronto（多伦多大学）；Vector Institute（向量研究所）；The Hospital for Sick Children（病童医院）
Despite the success of machine learning applications in science, industry, and society in general, many approaches are known to be non-robust, often relying on spurious correlations to make predictions. Spuriousness occurs when some features correlate with labels but are not causal; relying on such features prevents models from generalizing to unseen environments where such correlations break. In this work, we focus on image classification and propose two data generation processes to reduce spuriousness. Given human annotations of the subset of the features responsible (causal) for the labels …
C2RM-Seg: Causal Counterfactual Reasoning with Structural-Semantic Priors for Weakly Supervised Histopathological Tissue Segmentation
Guangxi Key Laboratory of Image and Graphic Intelligent Processing, Guilin University of Electronic Technology（广西图像图形智能处理重点实验室，桂林电子科技大学）；International Joint Research Laboratory of Spatio-temporal Information and Intelligent Location Services, Guilin University of Electronic Technology（时空信息与智能定位服务国际联合研究实验室，桂林电子科技大学）；School of Computer Science and Information Security, Guilin University of Electronic Technology（桂林电子科技大学计算机与信息安全学院）
Center for Data Science, Peking University（北京大学数据科学中心）；Deepwise AI lab（Deepwise AI实验室）；Dept. of Computer Science, Peking University（北京大学计算机科学系）；Microsoft Research Asia（微软亚洲研究院）；Center on Frontiers of Computing Studies, Dept. of Computer Science, Peking University（北京大学计算机科学系前沿计算研究中心）
Mammogram benign or malignant classification with only image-level labels is challenging due to the absence of lesion annotations. Motivated by the symmetric prior that the lesions on one side of breasts rarely appear in the corresponding areas on the other side, given a diseased image, we can explore a counterfactual problem that how would the features have behaved if there were no lesions in the image, so as to identify the lesion areas. We derive a new theoretical result for counterfactual generation based on the symmetric prior. By building a causal model that entails such a prior for bila…
The University of Edinburgh（爱丁堡大学）；Canon Medical Research Europe（佳能医疗欧洲研究院）；University of Glasgow（格拉斯哥大学）；The Alan Turing Institute（艾伦·图灵研究所）
Reducing the requirement for densely annotated masks in medical image segmentation is important due to cost constraints. In this paper, we consider the problem of inferring pixel-level predictions of brain lesions by only using image-level labels for training. By leveraging recent advances in generative diffusion probabilistic models (DPM), we synthesize counterfactuals of "How would a patient appear if X pathology was not present?". The difference image between the observed patient state and the healthy counterfactual can be used for inferring the location of pathology. We generate counterfac…
COMSATS University Islamabad（伊斯兰堡COMSATS大学）；National Center of Artificial Intelligence（国家人工智能中心）；University of Strasbourg（斯特拉斯堡大学）；CNRS（法国国家科学研究中心）；Ecole de technologie superieure（高等技术学院）
The image-based diagnosis is now a vital aspect of modern automation assisted diagnosis. To enable models to produce pixel-level diagnosis, pixel-level ground-truth labels are essentially required. However, since it is often not straight forward to obtain the labels in many application domains such as in medical image, classification-based approaches have become the de facto standard to perform the diagnosis. Though they can identify class-salient regions, they may not be useful for diagnosis where capturing all of the evidences is important requirement. Alternatively, a counterfactual explana…
Existing studies on disease diagnostic models focus either on diagnostic model learning for performance improvement or on the visual explanation of a trained diagnostic model. We propose a novel learn-explain-reinforce (LEAR) framework that unifies diagnostic model learning, visual explanation generation (explanation unit), and trained diagnostic model reinforcement (reinforcement unit) guided by the visual explanation. For the visual explanation, we generate a counterfactual map that transforms an input sample to be identified as an intended target label. For example, a counterfactual map can…
Reinforcing Pre-trained Models Using Counterfactual Images
Hokkaido University（北海道大学）；Graduate School of Information Science and Technology, Hokkaido University（北海道大学信息科学与技术研究院）；Faculty of Information Science and Technology, Hokkaido University（北海道大学信息科学与技术学院）
University of Pittsburgh（匹兹堡大学）；Carnegie Mellon University（卡内基梅隆大学）；University of Pittsburgh Medical School（匹兹堡大学医学院）
We propose a BlackBox Counterfactual Explainer, designed to explain image classification models for medical applications. Classical approaches (e.g., saliency maps) that assess feature importance do not explain "how" imaging features in important anatomical regions are relevant to the classification decision. Our framework explains the decision for a target class by gradually "exaggerating" the semantic effect of the class in a query image. We adopted a Generative Adversarial Network (GAN) to generate a progressive set of perturbations to a query image, such that the classification decision ch…
Beijing Institute of Technology（北京理工大学）；ETH Zurich（苏黎世联邦理工学院）；University of Macau（澳门大学）；University of Technology Sydney（悉尼科技大学）
Since the rise of vision-language navigation (VLN), great progress has been made in instruction following -- building a follower to navigate environments under the guidance of instructions. However, far less attention has been paid to the inverse task: instruction generation -- learning a speaker~to generate grounded descriptions for navigation routes. Existing VLN methods train a speaker independently and often treat it as a data augmentation tool to strengthen the follower while ignoring rich cross-task relations. Here we describe an approach that learns the two tasks simultaneously and expl…
Knowledge-Aligned Counterfactual-Enhancement Diffusion Perception for Unsupervised Cross-Domain Visual Emotion Recognition
The Laboratory of Intelligent Collaborative Computing of UESTC（UESTC智能协同计算实验室）；Ubiquitous Intelligence and Trusted Services Key Laboratory of Sichuan Province（四川省 ubiquitous intelligence and trusted services 重点实验室）；Faculty of Information Technology, Monash University（墨尔本大学信息技术学院）
Do Vision Models Learn Physical Constraints or Rendering Shortcuts? A Counterfactual Benchmark for Grounded Physical Consistency
Tri-institutional Center for Translational Research in Neuroimaging and Data Science (TReNDS)（转化神经影像与数据科学三机构中心）；Georgia State University（佐治亚州立大学）；Georgia Institute of Technology（佐治亚理工学院）；Emory University（埃默里大学）
