The ability of to explain neural network decisions goes hand in hand with their safe deployment. Several methods have been proposed to highlight features important for a given network decision. However, there is no consensus on how to measure effectiveness of these methods. We propose a new procedure for evaluating explanations. We use it to investigate visual explanations extracted from a range of possible sources in a neural network. We quantify the benefit of combining these sources and challenge a recent appeal for taking bias parameters into account. We support our conclusions with a general assessment of the impact of bias parameters in ImageNet classifiers
Interpretation and explanation of deep models is critical towards wide adoption of systems that rely on them. In this paper, we propose a novel scheme for both interpretation as well as explanation in which, given a pretrained model, we automatically identify internal features relevant for the set of classes considered by the model, without relying on additional annotations. We interpret the model through average visualizations of this reduced set of features. Then, at test time, we explain the network prediction by accompanying the predicted class label with supporting visualizations derived …
Most recent state-of-the-art Visual Question Answering (VQA) systems are opaque black boxes that are only trained to fit the answer distribution given the question and visual content. As a result, these systems frequently take shortcuts, focusing on simple visual concepts or question priors. This phenomenon becomes more problematic as the questions become complex that requires more reasoning and commonsense knowledge. To address this issue, we present a novel framework that uses explanations for competing answers to help VQA systems select the correct answer. By training on human textual expla…
Class activation map (CAM) has been widely studied for visual explanation of the internal working mechanism of convolutional neural networks. The key of existing CAM-based methods is to compute effective weights to combine activation maps in the target convolution layer. Existing gradient and score based weighting schemes have shown superiority in ensuring either the discriminability or faithfulness of the CAM, but they normally cannot excel in both properties. In this paper, we propose a novel CAM weighting scheme, named FD-CAM, to improve both the faithfulness and discriminability of the CAM…
Keen Eye（基恩眼公司）；LTCI, Télécom Paris, Institut Polytechnique de Paris（巴黎理工学院巴黎电信学院LTCI实验室）
Deep learning methods are widely used for medical applications to assist medical doctors in their daily routines. While performances reach expert's level, interpretability (highlight how and what a trained model learned and why it makes a specific decision) is the next important challenge that deep learning methods need to answer to be fully integrated in the medical field. In this paper, we address the question of interpretability in the context of whole slide images (WSI) classification. We formalize the design of WSI classification architectures and propose a piece-wise interpretability app…
MIT Media Lab（麻省理工学院媒体实验室）；MIT CSAIL（麻省理工学院计算机科学与人工智能实验室）
We present a randomized controlled trial for a model-in-the-loop regression task, with the goal of measuring the extent to which (1) good explanations of model predictions increase human accuracy, and (2) faulty explanations decrease human trust in the model. We study explanations based on visual saliency in an image-based age prediction task for which humans and learned models are individually capable but not highly proficient and frequently disagree. Our experimental design separates model quality from explanation quality, and makes it possible to compare treatments involving a variety of ex…
Mitigating Bias: Enhancing Image Classification by Improving Model Explanations
Amirkabir University of Technology（阿米尔卡比尔理工大学）；Okinawa Institute of Science and Technology（冲绳科学技术大学院大学）；Institute for Research in Fundamental Sciences（基础科学研究所）
