Neural network visualization techniques mark image locations by their relevancy to the network's classification. Existing methods are effective in highlighting the regions that affect the resulting classification the most. However, as we show, these methods are limited in their ability to identify the support for alternative classifications, an effect we name {\em the saliency bias} hypothesis. In this work, we integrate two lines of research: gradient-based methods and attribution-based methods, and develop an algorithm that provides per-class explainability. The algorithm back-projects the per pixel local influence, in a manner that is guided by the local attributions, while correcting for salient features that would otherwise bias the explanation. In an extensive battery of experiments, we demonstrate the ability of our methods to class-specific visualization, and not just the predicted label. Remarkably, the method obtains state of the art results in benchmarks that are commonly applied to gradient-based methods as well as in those that are employed mostly for evaluating attribution methods. Using a new unsupervised procedure, our method is also successful in demonstrating that self-supervised methods learn semantic information.
Institut de Robotica i Informàtica Industrial, CSIC-UPC（机器人工业研究所，CSIC-UPC）
Attribution map visualization has arisen as one of the most effective techniques to understand the underlying inference process of Convolutional Neural Networks. In this task, the goal is to compute an score for each image pixel related with its contribution to the final network output. In this paper, we introduce Disentangled Masked Backpropagation (DMBP), a novel gradient-based method that leverages on the piecewise linear nature of ReLU networks to decompose the model function into different linear mappings. This decomposition aims to disentangle the positive, negative and nuisance factors …
Deep learning is increasingly used in decision-making tasks. However, understanding how neural networks produce final predictions remains a fundamental challenge. Existing work on interpreting neural network predictions for images often focuses on explaining predictions for single images or neurons. As predictions are often computed from millions of weights that are optimized over millions of images, such explanations can easily miss a bigger picture. We present Summit, an interactive system that scalably and systematically summarizes and visualizes what features a deep learning model has lear…
University of Toronto（多伦多大学）；LG AI Research（LG AI研究院）
As an emerging field in Machine Learning, Explainable AI (XAI) has been offering remarkable performance in interpreting the decisions made by Convolutional Neural Networks (CNNs). To achieve visual explanations for CNNs, methods based on class activation mapping and randomized input sampling have gained great popularity. However, the attribution methods based on these techniques provide lower resolution and blurry explanation maps that limit their explanation power. To circumvent this issue, visualization based on various layers is sought. In this work, we collect visualization maps from multi…
The clear transparency of Deep Neural Networks (DNNs) is hampered by complex internal structures and nonlinear transformations along deep hierarchies. In this paper, we propose a new attribution method, Relative Sectional Propagation (RSP), for fully decomposing the output predictions with the characteristics of class-discriminative attributions and clear objectness. We carefully revisit some shortcomings of backpropagation-based attribution methods, which are trade-off relations in decomposing DNNs. We define hostile factor as an element that interferes with finding the attributions of the ta…
Charité – Universitätsmedizin Berlin（柏林夏里特医学院）；Freie Universität Berlin（柏林自由大学）；Humboldt-Universität zu Berlin（柏林洪堡大学）；Berlin Institute of Health（柏林健康研究所）；Bernstein Center for Computational Neuroscience（伯恩斯坦计算神经科学中心）
A major prerequisite for the application of machine learning models in clinical decision making is trust and interpretability. Current explainability studies in the neuroimaging community have mostly focused on explaining individual decisions of trained models, e.g. obtained by a convolutional neural network (CNN). Using attribution methods such as layer-wise relevance propagation or SHAP heatmaps can be created that highlight which regions of an input are more relevant for the decision than others. While this allows the detection of potential data set biases and can be used as a guide for a h…
Sungkyunkwan University（成均馆大学）；Seoul National University（首尔大学）；University of Cambridge（剑桥大学）；The Alan Turing Institute（艾伦·图灵研究所）
Neural network interpretation methods, particularly feature attribution methods, are known to be fragile with respect to adversarial input perturbations. To address this, several methods for enhancing the local smoothness of the gradient while training have been proposed for attaining \textit{robust} feature attributions. However, the lack of considering the normalization of the attributions, which is essential in their visualizations, has been an obstacle to understanding and improving the robustness of feature attribution methods. In this paper, we provide new insights by taking such normali…
Carney Institute for Brain Science, Brown University（布朗大学卡尼脑科学研究所）；Sorbonne Université（索邦大学）；CNRS（法国国家科学研究中心）；Artificial and Natural Intelligence Toulouse Institute, Université de Toulouse（图卢兹大学人工智能与自然智能研究所）；Institut de Recherche Technologique Saint-Exupery（圣埃克苏佩里技术研究所）；Innovation & Research Division, SNCF（法国国家铁路公司创新与研究部）；Scalian（Scalian公司）
Attribution methods, which employ heatmaps to identify the most influential regions of an image that impact model decisions, have gained widespread popularity as a type of explainability method. However, recent research has exposed the limited practical value of these methods, attributed in part to their narrow focus on the most prominent regions of an image -- revealing "where" the model looks, but failing to elucidate "what" the model sees in those areas. In this work, we try to fill in this gap with CRAFT -- a novel approach to identify both "what" and "where" by generating concept-based ex…
Korea University（高丽大学）；Tel Aviv University（特拉维夫大学）；KAIST（韩国科学技术院）；Facebook AI Research（脸书人工智能研究院）
As Deep Neural Networks (DNNs) have demonstrated superhuman performance in a variety of fields, there is an increasing interest in understanding the complex internal mechanisms of DNNs. In this paper, we propose Relative Attributing Propagation (RAP), which decomposes the output predictions of DNNs with a new perspective of separating the relevant (positive) and irrelevant (negative) attributions according to the relative influence between the layers. The relevance of each neuron is identified with respect to its degree of contribution, separated into positive and negative, while preserving th…
Contrastive Integrated Gradients: A Feature Attribution-Based Method for Explaining Whole Slide Image Classification
ECE Department, University of Houston（德克萨斯大学休斯顿分校电子与计算机工程系）；Information Technology, HCMC University of Technology and Education（胡志明市技术与教育大学信息技术系）；VN-UK Institute for Research and Executive Education, The University of Da Nang（越南-英国研究与管理教育研究所，丹那大学）；Ho Chi Minh City University of Science, Vietnam National University（胡志明市科学大学，越南国家大学）；Computational Biology Department, Carnegie Mellon University（卡内基梅隆大学计算生物学系）
Unlocking Feature Visualization for Deeper Networks with MAgnitude Constrained Optimization
Artificial and Natural Intelligence Toulouse Institute（图卢兹人工与自然智能研究所）；Innovation & Research Division, SNCF（法国国家铁路公司创新与研究部）；Institut de Recherche Technologique Saint-Exupery（圣埃克苏佩里技术研究院）；Carney Institute for Brain Science, Brown University（布朗大学卡尼脑科学研究所）；ELLIS Alicante（欧洲学习与智能系统研究所阿利坎特分部）
University of Toulouse（图卢兹大学）；IRIT/INP-ENSEEIHT Toulouse（图卢兹国立理工学院ENSEEIHT校区信息与电信研究所）；CESBIO（空间研究中心）；CNES（法国国家空间研究中心）；CNRS（法国国家科学研究中心）；INRA（法国国家农业研究院）；IRD（法国发展研究所）；UPS（图卢兹第三大学）
Supervised classification and representation learning are two widely used classes of methods to analyze multivariate images. Although complementary, these methods have been scarcely considered jointly in a hierarchical modeling. In this paper, a method coupling these two approaches is designed using a matrix cofactorization formulation. Each task is modeled as a factorization matrix problem and a term relating both coding matrices is then introduced to drive an appropriate coupling. The link can be interpreted as a clustering operation over a low-dimensional representation vectors. The attribu…
Convolutional Neural Networks (CNN) have become state-of-the-art in the field of image classification. However, not everything is understood about their inner representations. This paper tackles the interpretability and explainability of the predictions of CNNs for multi-class classification problems. Specifically, we propose a novel visualization method of pixel-wise input attribution called Softmax-Gradient Layer-wise Relevance Propagation (SGLRP). The proposed model is a class discriminate extension to Deep Taylor Decomposition (DTD) using the gradient of softmax to back propagate the relev…
