Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications
Fraunhofer Heinrich Hertz Institute（弗劳恩霍夫海因里希·赫兹研究所）；BIFOLD – Berlin Institute for the Foundations of Learning and Data（BIFOLD——柏林学习与数据基础研究所）；Technische Universität Berlin（柏林工业大学）；Korea University（高丽大学）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）
Wojciech Samek, Grégoire Montavon, Sebastian Lapuschkin, Christopher J. Anders, Klaus-Robert Müller
With the broader and highly successful usage of machine learning in industry and the sciences, there has been a growing demand for Explainable AI. Interpretability and explanation methods for gaining a better understanding about the problem solving abilities and strategies of nonlinear Machine Learning, in particular, deep neural networks, are therefore receiving increased attention. In this work we aim to (1) provide a timely overview of this active emerging field, with a focus on 'post-hoc' explanations, and explain its theoretical foundations, (2) put interpretability algorithms to a test both from a theory and comparative evaluation perspective using extensive simulations, (3) outline best practice aspects i.e. how to best include interpretation methods into the standard usage of machine learning and (4) demonstrate successful usage of explainable AI in a representative selection of application scenarios. Finally, we discuss challenges and possible future directions of this exciting foundational field of machine learning.
Interpretability is an important property for visual models as it helps researchers and users understand the internal mechanism of a complex model. However, generating semantic explanations about the learned representation is challenging without direct supervision to produce such explanations. We propose a general framework, Latent Visual Semantic Explainer (LaViSE), to teach any existing convolutional neural network to generate text descriptions about its own latent representations at the filter level. Our method constructs a mapping between the visual and semantic spaces using generic image …
University of Texas at Austin（德克萨斯大学奥斯汀分校）；Carnegie Mellon University（卡内基梅隆大学）；Google Strategic Technologies（谷歌战略技术部）；Marianas Labs（马里亚纳实验室）
Understanding a user's motivations provides valuable information beyond the ability to recommend items. Quite often this can be accomplished by perusing both ratings and review texts, since it is the latter where the reasoning for specific preferences is explicitly expressed. Unfortunately matrix factorization approaches to recommendation result in large, complex models that are difficult to interpret and give recommendations that are hard to clearly explain to users. In contrast, in this paper, we attack this problem through succinct additive co-clustering. We devise a novel Bayesian techniqu…
Fraunhofer Heinrich Hertz Institute（弗劳恩霍夫海因里希·赫兹研究所）；Singapore Institute of Technology（新加坡理工大学）；University of Oslo（奥斯陆大学）；Berlin Institute for the Foundations of Learning and Data（柏林学习与数据基础研究所）
Explainable Artificial Intelligence (XAI) is an emerging research field bringing transparency to highly complex and opaque machine learning (ML) models. Despite the development of a multitude of methods to explain the decisions of black-box classifiers in recent years, these tools are seldomly used beyond visualization purposes. Only recently, researchers have started to employ explanations in practice to actually improve models. This paper offers a comprehensive overview over techniques that apply XAI practically for improving various properties of ML models, and systematically categorizes th…
Deep generative models are a class of techniques that train deep neural networks to model the distribution of training samples. Research has fragmented into various interconnected approaches, each of which make trade-offs including run-time, diversity, and architectural restrictions. In particular, this compendium covers energy-based models, variational autoencoders, generative adversarial networks, autoregressive models, normalizing flows, in addition to numerous hybrid approaches. These techniques are compared and contrasted, explaining the premises behind each and how they are interrelated,…
Beyond Fidelity: Explaining Vulnerability Localization of Learning-based Detectors
Peking University（北京大学）；University of Alberta（阿尔伯塔大学）；Huazhong University of Science and Technology（华中科技大学）；University of Queensland（昆士兰大学）；University of New South Wales（新南威尔士大学）
Deep neural networks are increasingly used in natural language processing (NLP) models. However, the need to interpret and explain the results from complex algorithms are limiting their widespread adoption in regulated industries such as banking. There has been recent work on interpretability of machine learning algorithms with structured data. But there are only limited techniques for NLP applications where the problem is more challenging due to the size of the vocabulary, high-dimensional nature, and the need to consider textual coherence and language structure. This paper develops a methodo…
Dalle Molle Institute for Artificial Intelligence（达勒·莫尔人工智能研究所）；University of Southern Switzerland（南瑞士大学）；University of Applied Sciences and Arts of Southern Switzerland（南瑞士应用科学与艺术大学）；Information Systems and Networking Institute（信息系统与网络研究所）
ChatGPT has the ability to generate grammatically flawless and seemingly-human replies to different types of questions from various domains. The number of its users and of its applications is growing at an unprecedented rate. Unfortunately, use and abuse come hand in hand. In this paper, we study whether a machine learning model can be effectively trained to accurately distinguish between original human and seemingly human (that is, ChatGPT-generated) text, especially when this text is short. Furthermore, we employ an explainable artificial intelligence framework to gain insight into the reaso…
Collaborative Robotics and Intelligent Systems (CoRIS) Institute（协作机器人与智能系统研究所）；Oregon State University（俄勒冈州立大学）
In many object recognition applications, the set of possible categories is an open set, and the deployed recognition system will encounter novel objects belonging to categories unseen during training. Detecting such "novel category" objects is usually formulated as an anomaly detection problem. Anomaly detection algorithms for feature-vector data identify anomalies as outliers, but outlier detection has not worked well in deep learning. Instead, methods based on the computed logits of visual object classifiers give state-of-the-art performance. This paper proposes the Familiarity Hypothesis th…
University of Southern California（南加州大学）；Facebook AI（脸书人工智能研究院）
Recommendation is a prevalent application of machine learning that affects many users; therefore, it is important for recommender models to be accurate and interpretable. In this work, we propose a method to both interpret and augment the predictions of black-box recommender systems. In particular, we propose to interpret feature interactions from a source recommender model and explicitly encode these interactions in a target recommender model, where both source and target models are black-boxes. By not assuming the structure of the recommender system, our approach can be used in general setti…
University of Michigan（密歇根大学）；Stanford University（斯坦福大学）；NTT Research, Inc.（NTT研究院）
Inspired by BatchNorm, there has been an explosion of normalization layers in deep learning. Recent works have identified a multitude of beneficial properties in BatchNorm to explain its success. However, given the pursuit of alternative normalization layers, these properties need to be generalized so that any given layer's success/failure can be accurately predicted. In this work, we take a first step towards this goal by extending known properties of BatchNorm in randomly initialized deep neural networks (DNNs) to several recently proposed normalization layers. Our primary findings follow: (…
A Survey on Explainable Reinforcement Learning: Concepts, Algorithms, Challenges
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算与数据科学学院）；School of Software Technology, Zhejiang University（浙江大学软件技术学院）；State Key Laboratory of Blockchain and Security, Zhejiang University（浙江大学区块链与安全国家重点实验室）
Optical spectral-temporal signatures extracted from videos of explosions provide information for identifying characteristics of the corresponding explosive devices. Currently, the identification is done using heuristic algorithms and direct subject matter expert review. An improvement in predictive performance may be obtained by using machine learning, but this application lends itself to high consequence national security decisions, so it is not only important to provide high accuracy but clear explanations for the predictions to garner confidence in the model. While much work has been done t…
Google Research（谷歌研究院）；Grammarly；Megagon Labs（Megagon实验室）；University of Illinois at Chicago（伊利诺伊大学芝加哥分校）
Customer reviews are vital for making purchasing decisions in the Information Age. Such reviews can be automatically summarized to provide the user with an overview of opinions. In this tutorial, we present various aspects of opinion summarization that are useful for researchers and practitioners. First, we will introduce the task and major challenges. Then, we will present existing opinion summarization solutions, both pre-neural and neural. We will discuss how summarizers can be trained in the unsupervised, few-shot, and supervised regimes. Each regime has roots in different machine learning…
CoxSE: Exploring the Potential of Self-Explaining Neural Networks with Cox Proportional Hazards Model for Survival Analysis
Center for Applied Intelligent Systems Research (CAISR)（应用智能系统研究中心）；Halmstad University（哈马斯特德大学）；Centre for Environmental and Climate Science（环境与气候科学中心）；Lund University（吕勒奥大学）；Research Institutes of Sweden (RISE)（瑞典研究机构）
Online detection and infographic explanation of spam reviews with data drift adaptation
University of Vigo（维戈大学）；Universidade Portucalense（波尔图大学）；ISEP, Polytechnic of Porto（波尔图理工学院ISEP）；Institute for Systems and Computer Engineering, Technology and Science（系统与计算机工程、技术与科学研究所）
University of Applied Sciences Fulda（富尔达应用科学大学）；Montréal University (UdeM)（蒙特利尔大学）；Mila（米拉研究所）
Continual Learning (CL, sometimes also termed incremental learning) is a flavor of machine learning where the usual assumption of stationary data distribution is relaxed or omitted. When naively applying, e.g., DNNs in CL problems, changes in the data distribution can cause the so-called catastrophic forgetting (CF) effect: an abrupt loss of previous knowledge. Although many significant contributions to enabling CL have been made in recent years, most works address supervised (classification) problems. This article reviews literature that study CL in other settings, such as learning with reduc…
Interpreting "Interpretability" and Explaining "Explainability" in Machine Learning in Physics
Department of Physics, University of Cincinnati（俄亥俄州辛辛那提大学物理系）；Hamburger Sternwarte, Universität Hamburg（汉堡大学汉堡天文台）；Center for Theoretical Physics – a Leinweber Institute, Massachusetts Institute of Technology（麻省理工学院理论物理中心——莱因韦伯研究所）；Institut des Hautes Études Scientifiques（高等科学研究院）；Institut de Physique Théorique, CEA Paris-Saclay（巴黎-萨克雷CEA理论物理研究所）；NSF Institute for Artificial Intelligence and Fundamental Interactions（国家科学基金会人工智能与基本相互作用研究所）
