Although deep reinforcement learning agents have produced impressive results in many domains, their decision making is difficult to explain to humans. To address this problem, past work has mainly focused on explaining why an action was chosen in a given state. A different type of explanation that is useful is a counterfactual, which deals with "what if?" scenarios. In this work, we introduce the concept of a counterfactual state to help humans gain a better understanding of what would need to change (minimally) in an Atari game image for the agent to choose a different action. We introduce a novel method to create counterfactual states from a generative deep learning architecture. In addition, we evaluate the effectiveness of counterfactual states on human participants who are not machine learning experts. Our user study results suggest that our generated counterfactual states are useful in helping non-expert participants gain a better understanding of an agent's decision making process.
Counterfactual explanations, which deal with "why not?" scenarios, can provide insightful explanations to an AI agent's behavior. In this work, we focus on generating counterfactual explanations for deep reinforcement learning (RL) agents which operate in visual input environments like Atari. We introduce counterfactual state explanations, a novel example-based approach to counterfactual explanations based on generative deep learning. Specifically, a counterfactual state illustrates what minimal change is needed to an Atari game image such that the agent chooses a different action. We also eva…
Using Generative Adversarial Nets on Atari Games for Feature Extraction in Deep Reinforcement Learning
Middle East Technical University（中东技术大学）
深度强化学习（Deep Reinforcement Learning，DRL）已成功应用于机器人导航和自动视频游戏等多个研究领域。然而，这些方法需要过多的计算和与环境的交互，因此需要提高样本效率。这一要求的主要原因是稀疏和延迟的奖励不能为深度神经网络的表示学习提供有效的监督。在这项研究中，近端策略优化（PPO）算法与生成对抗网络（GANs）增强，通过强制网络学习有效的表示而不依赖于稀疏和延迟的奖励作为监督来提高样本效率。结果表明，通过联合训练DRL代理和GAN代理，可以获得更高的性能。- Derin Pekistirmeli Oakme，robot navigasyonu ve otomatikelestirilmis video oyunu oaku a gibi arastirma alanlarinda basariyla uygulanmaktadir.此外，我们还可以看到，由于气候变化和环境变化，我们的工作人员无法确定他们的工作是否正常。但是，在一个人的生活中，人们对自己的生活和生活的看法是，他们的生活和生活是如此的丰富多彩，以至于他们的生活和生活都变得如此的艰难。但据了解，Uretici Cekismeli Aglar（UCA）算法的优化政策方案将使目标明确，并使目标明确或使目标明确，从而使目标明确。其他的声音算法也可以用其他的方法来实现。
Australian Artificial Intelligence Institute, University of Technology Sydney（悉尼科技大学澳大利亚人工智能研究院）
Imitation learning in a high-dimensional environment is challenging. Most inverse reinforcement learning (IRL) methods fail to outperform the demonstrator in such a high-dimensional environment, e.g., Atari domain. To address this challenge, we propose a novel reward learning module to generate intrinsic reward signals via a generative model. Our generative method can perform better forward state transition and backward action encoding, which improves the module's dynamics modeling ability in the environment. Thus, our module provides the imitation agent both the intrinsic intention of the dem…
We present a novel generative method for producing unseen and plausible counterfactual examples for reinforcement learning (RL) agents based upon outcome variables that characterize agent behavior. Our approach uses a variational autoencoder to train a latent space that jointly encodes information about the observations and outcome variables pertaining to an agent's behavior. Counterfactuals are generated using traversals in this latent space, via gradient-driven updates as well as latent interpolations against cases drawn from a pool of examples. These include updates to raise the likelihood …
Mila（米拉研究所）；Université de Montréal（蒙特利尔大学）；Microsoft Research（微软研究院）
State representation learning, or the ability to capture latent generative factors of an environment, is crucial for building intelligent agents that can perform a wide variety of tasks. Learning such representations without supervision from rewards is a challenging open problem. We introduce a method that learns state representations by maximizing mutual information across spatially and temporally distinct features of a neural encoder of the observations. We also introduce a new benchmark based on Atari 2600 games where we evaluate representations based on how well they capture the ground tru…
The prevalence of machine learning models in various industries has led to growing demands for model interpretability and for the ability to provide meaningful recourse to users. For example, patients hoping to improve their diagnoses or loan applicants seeking to increase their chances of approval. Counterfactuals can help in this regard by identifying input perturbations that would result in more desirable prediction outcomes. Meaningful counterfactuals should be able to achieve the desired outcome, but also be realistic, actionable, and efficient to compute. Current approaches achieve desir…
Counterfactual instances offer human-interpretable insight into the local behaviour of machine learning models. We propose a general framework to generate sparse, in-distribution counterfactual model explanations which match a desired target prediction with a conditional generative model, allowing batches of counterfactual instances to be generated with a single forward pass. The method is flexible with respect to the type of generative model used as well as the task of the underlying predictive model. This allows straightforward application of the framework to different modalities such as ima…
Berlin Institute of Technology (TU Berlin)（柏林工业大学）；Korea University（高丽大学）；Max Planck Institut für Informatik（马克斯·普朗克信息学研究所）；Google Research（谷歌研究院）
Counterfactuals can explain classification decisions of neural networks in a human interpretable way. We propose a simple but effective method to generate such counterfactuals. More specifically, we perform a suitable diffeomorphic coordinate transformation and then perform gradient ascent in these coordinates to find counterfactuals which are classified with great confidence as a specified target class. We propose two methods to leverage generative models to construct such suitable coordinate systems that are either exactly or approximately diffeomorphic. We analyze the generation process the…
DoFlow: Flow-based Generative Models for Interventional and Counterfactual Forecasting on Time Series
H. Milton Stewart School of Industrial and Systems Engineering（H. Milton Stewart工业与系统工程学院）；Georgia Institute of Technology（佐治亚理工学院）；Northwestern–Argonne Institute for Scientific and Engineering Excellence（西北-阿贡科学与工程卓越研究所）
A machine learning model, under the influence of observed or unobserved confounders in the training data, can learn spurious correlations and fail to generalize when deployed. For image classifiers, augmenting a training dataset using counterfactual examples has been empirically shown to break spurious correlations. However, the counterfactual generation task itself becomes more difficult as the level of confounding increases. Existing methods for counterfactual generation under confounding consider a fixed set of interventions (e.g., texture, rotation) and are not flexible enough to capture d…
With the ongoing rise of machine learning, the need for methods for explaining decisions made by artificial intelligence systems is becoming a more and more important topic. Especially for image classification tasks, many state-of-the-art tools to explain such classifiers rely on visual highlighting of important areas of the input data. Contrary, counterfactual explanation systems try to enable a counterfactual reasoning by modifying the input image in a way such that the classifier would have made a different prediction. By doing so, the users of counterfactual explanation systems are equippe…
With the wide use of deep neural networks (DNN), model interpretability has become a critical concern, since explainable decisions are preferred in high-stake scenarios. Current interpretation techniques mainly focus on the feature attribution perspective, which are limited in indicating why and how particular explanations are related to the prediction. To this end, an intriguing class of explanations, named counterfactuals, has been developed to further explore the "what-if" circumstances for interpretation, and enables the reasoning capability on black-box models. However, generating counter…
Generative Flows for Potential Outcomes and Counterfactuals
School of Industrial and Systems Engineering（工业与系统工程学院）；Georgia Institute of Technology（佐治亚理工学院）；Elmore Family School of Electrical and Computer Engineering（埃洛姆家族电气与计算机工程学院）；Purdue University（普渡大学）
QUCE: The Minimisation and Quantification of Path-Based Uncertainty for Generative Counterfactual Explanations
School of Mathematics and Computer Science（数学与计算机科学学院）；Swansea University（斯旺西大学）；Department of Finance, Insurance and Real Estate（金融、保险与房地产系）；Université Laval（拉瓦尔大学）；Lee Kong Chian School of Medicine（李科珍医学院）；School of Computer Science and Engineering（计算机科学与工程学院）；Nanyang Technological University（南洋理工大学）
We describe Countersynth, a conditional generative model of diffeomorphic deformations that induce label-driven, biologically plausible changes in volumetric brain images. The model is intended to synthesise counterfactual training data augmentations for downstream discriminative modelling tasks where fidelity is limited by data imbalance, distributional instability, confounding, or underspecification, and exhibits inequitable performance across distinct subpopulations. Focusing on demographic attributes, we evaluate the quality of synthesized counterfactuals with voxel-based morphometry, clas…
University of Virginia（弗吉尼亚大学）；Bytedance AI Lab（字节跳动人工智能实验室）；J.P. Morgan AI Research（摩根大通人工智能研究院）
Counterfactual explanations promote explainability in machine learning models by answering the question "how should an input instance be perturbed to obtain a desired predicted label?". The comparison of this instance before and after perturbation can enhance human interpretation. Most existing studies on counterfactual explanations are limited in tabular data or image data. In this work, we study the problem of counterfactual explanation generation on graphs. A few studies have explored counterfactual explanations on graphs, but many challenges of this problem are still not well-addressed: 1)…
Massachusetts Institute of Technology（麻省理工学院）；Microsoft Research（微软研究院）
Recent advances in probabilistic generative modeling have motivated learning Structural Causal Models (SCM) from observational datasets using deep conditional generative models, also known as Deep Structural Causal Models (DSCM). If successful, DSCMs can be utilized for causal estimation tasks, e.g., for answering counterfactual queries. In this work, we warn practitioners about non-identifiability of counterfactual inference from observational data, even in the absence of unobserved confounding and assuming known causal structure. We prove counterfactual identifiability of monotonic generatio…
Variational autoencoders (VAEs) and other generative methods have garnered growing interest not just for their generative properties but also for the ability to dis-entangle a low-dimensional latent variable space. However, few existing generative models take causality into account. We propose a new decoder based framework named the Causal Counterfactual Generative Model (CCGM), which includes a partially trainable causal layer in which a part of a causal model can be learned without significantly impacting reconstruction fidelity. By learning the causal relationships between image semantic la…
Modeling Matches as Language: A Generative Transformer Approach for Counterfactual Player Valuation in Football
Department of Artificial Intelligence, University of Seoul, Seoul, Republic of Korea（首尔大学人工智能系）；Institute for Sports and Preventive Medicine, Saarland University, Saarbrücken, Germany（萨尔兰州预防医学研究所）；Chair for Sports Analytics, Saarland University（萨尔兰州体育分析教授职位）
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；University of Tübingen（蒂宾根大学）
Neural networks are prone to learning shortcuts -- they often model simple correlations, ignoring more complex ones that potentially generalize better. Prior works on image classification show that instead of learning a connection to object shape, deep classifiers tend to exploit spurious correlations with low-level texture or the background for solving the classification task. In this work, we take a step towards more robust and interpretable classifiers that explicitly expose the task's causal structure. Building on current advances in deep generative modeling, we propose to decompose the im…
Okayama University（冈山大学）；Japan Science and Technology Agency（日本科学技术振兴机构）；Rikkyo University（立教大学）；National Institute for Physiological Sciences（生理学研究所）；Araya Inc.（Araya 公司）；Keio University（庆应义塾大学）
Deep neural networks (DNNs) can accurately decode task-related information from brain activations. However, because of the nonlinearity of the DNN, the decisions made by DNNs are hardly interpretable. One of the promising approaches for explaining such a black-box system is counterfactual explanation. In this framework, the behavior of a black-box system is explained by comparing real data and realistic synthetic data that are specifically generated such that the black-box system outputs an unreal outcome. Here we introduce a novel generative DNN (counterfactual activation generator, CAG) that…
Collaborative Robotics and Intelligent Systems (CoRIS) Institute（协作机器人与智能系统（CoRIS）研究所）；Oregon State University（俄勒冈州立大学）
CounterFactual (CF) visual explanations try to find images similar to the query image that change the decision of a vision system to a specified outcome. Existing methods either require inference-time optimization or joint training with a generative adversarial model which makes them time-consuming and difficult to use in practice. We propose a novel approach, Cycle-Consistent Counterfactuals by Latent Transformations (C3LT), which learns a latent transformation that automatically generates visual CFs by steering in the latent space of generative models. Our method uses cycle consistency betwe…
