Many proposed methods for explaining machine learning predictions are in fact challenging to understand for nontechnical consumers. This paper builds upon an alternative consumer-driven approach called TED that asks for explanations to be provided in training data, along with target labels. Using semi-synthetic data from credit approval and employee retention applications, experiments are conducted to investigate some practical considerations with TED, including its performance with different classification algorithms, varying numbers of explanations, and variability in explanations. A new algorithm is proposed to handle the case where some training examples do not have explanations. Our results show that TED is robust to increasing numbers of explanations, noisy explanations, and large fractions of missing explanations, thus making advances toward its practical deployment.
University of Michigan（密歇根大学）；University of Washington（华盛顿大学）
Recently, interpretable models called self-explaining models (SEMs) have been proposed with the goal of providing interpretability robustness. We evaluate the interpretability robustness of SEMs and show that explanations provided by SEMs as currently proposed are not robust to adversarial inputs. Specifically, we successfully created adversarial inputs that do not change the model outputs but cause significant changes in the explanations. We find that even though current SEMs use stable co-efficients for mapping explanations to output labels, they do not consider the robustness of the first s…
Transparency is a fundamental requirement for decision making systems when these should be deployed in the real world. It is usually achieved by providing explanations of the system's behavior. A prominent and intuitive type of explanations are counterfactual explanations. Counterfactual explanations explain a behavior to the user by proposing actions -- as changes to the input -- that would cause a different (specified) behavior of the system. However, such explanation methods can be unstable with respect to small changes to the input -- i.e. even a small change in the input can lead to huge …
J.P. Morgan AI Research（摩根大通人工智能研究院）；University of Maryland（马里兰大学）
There exist several methods that aim to address the crucial task of understanding the behaviour of AI/ML models. Arguably, the most popular among them are local explanations that focus on investigating model behaviour for individual instances. Several methods have been proposed for local analysis, but relatively lesser effort has gone into understanding if the explanations are robust and accurately reflect the behaviour of underlying models. In this work, we present a survey of the works that analysed the robustness of two classes of local explanations (feature importance and counterfactual ex…
University of Washington（华盛顿大学）；Carnegie Mellon University（卡内基梅隆大学）；UCLA（加州大学洛杉矶分校）；Google Research（谷歌研究院）
Feature based explanations, that provide importance of each feature towards the model prediction, is arguably one of the most intuitive ways to explain a model. In this paper, we establish a novel set of evaluation criteria for such feature based explanations by robustness analysis. In contrast to existing evaluations which require us to specify some way to "remove" features that could inevitably introduces biases and artifacts, we make use of the subtler notion of smaller adversarial perturbations. By optimizing towards our proposed evaluation criteria, we obtain new explanations that are loo…
The use of counterfactual explanations (CFXs) is an increasingly popular explanation strategy for machine learning models. However, recent studies have shown that these explanations may not be robust to changes in the underlying model (e.g., following retraining), which raises questions about their reliability in real-world applications. Existing attempts towards solving this problem are heuristic, and the robustness to model changes of the resulting CFXs is evaluated with only a small number of retrained models, failing to provide exhaustive guarantees. To remedy this, we propose Δ-robustness…
Using machine learning in high-stakes applications often requires predictions to be accompanied by explanations comprehensible to the domain user, who has ultimate responsibility for decisions and outcomes. Recently, a new framework for providing explanations, called TED, has been proposed to provide meaningful explanations for predictions. This framework augments training data to include explanations elicited from domain users, in addition to features and labels. This approach ensures that explanations for predictions are tailored to the complexity expectations and domain knowledge of the con…
Explainability in machine learning has become incredibly important as machine learning-powered systems become ubiquitous and both regulation and public sentiment begin to demand an understanding of how these systems make decisions. As a result, a number of explanation methods have begun to receive widespread adoption. This work summarizes, compares, and contrasts three popular explanation methods: LIME, SmoothGrad, and SHAP. We evaluate these methods with respect to: robustness, in the sense of sample complexity and stability; understandability, in the sense that provided explanations are cons…
University of Texas at Austin（德克萨斯大学奥斯汀分校）；CognitiveScale
As artificial intelligence plays an increasingly important role in our society, there are ethical and moral obligations for both businesses and researchers to ensure that their machine learning models are designed, deployed, and maintained responsibly. These models need to be rigorously audited for fairness, robustness, transparency, and interpretability. A variety of methods have been developed that focus on these issues in isolation, however, managing these methods in conjunction with model development can be cumbersome and timeconsuming. In this paper, we introduce a unified and model-agnos…
While machine-learning algorithms have demonstrated a strong ability in detecting Android malware, they can be evaded by sparse evasion attacks crafted by injecting a small set of fake components, e.g., permissions and system calls, without compromising intrusive functionality. Previous work has shown that, to improve robustness against such attacks, learning algorithms should avoid overemphasizing few discriminant features, providing instead decisions that rely upon a large subset of components. In this work, we investigate whether gradient-based attribution methods, used to explain classifie…
When Can You Trust Your Explanations? A Robustness Analysis on Feature Importances
Department of Mathematics, Informatics and Geosciences（数学、信息学与地球科学系）；University of Trieste（特伦托大学）；The Abdus Salam International Center for Theoretical Physics（阿布杜斯·萨拉姆国际理论物理中心）；Assicurazioni Generali Spa（安联保险公司）
The Alan Turing Institute（阿兰·图灵研究所）；University of Cambridge（剑桥大学）；Accenture Labs（埃森哲实验室）
Post-hoc explanation methods are used with the intent of providing insights about neural networks and are sometimes said to help engender trust in their outputs. However, popular explanations methods have been found to be fragile to minor perturbations of input features or model parameters. Relying on constraint relaxation techniques from non-convex optimization, we develop a method that upper-bounds the largest change an adversary can make to a gradient-based explanation via bounded manipulation of either the input features or model parameters. By propagating a compact input or parameter set …
IBM Research Zurich（IBM苏黎世研究院）；École Polytechnique Fédérale de Lausanne (EPFL)（洛桑联邦理工学院）
Explanations are crucial parts of deep neural network (DNN) classifiers. In high stakes applications, faithful and robust explanations are important to understand and gain trust in DNN classifiers. However, recent work has shown that state-of-the-art attribution methods in text classifiers are susceptible to imperceptible adversarial perturbations that alter explanations significantly while maintaining the correct prediction outcome. If undetected, this can critically mislead the users of DNNs. Thus, it is crucial to understand the influence of such adversarial perturbations on the networks' e…
University of Waterloo（滑铁卢大学）；Harvard University（哈佛大学）；Carnegie Mellon University（卡内基梅隆大学）
As machine learning black boxes are increasingly being deployed in critical domains such as healthcare and criminal justice, there has been a growing emphasis on developing techniques for explaining these black boxes in a post hoc manner. In this work, we analyze two popular post hoc interpretation techniques: SmoothGrad which is a gradient based method, and a variant of LIME which is a perturbation based method. More specifically, we derive explicit closed form expressions for the explanations output by these two methods and show that they both converge to the same explanation in expectation,…
Indian Institute of Technology, Hyderabad（印度理工学院海得拉巴分校）
Explainability has been widely stated as a cornerstone of the responsible and trustworthy use of machine learning models. With the ubiquitous use of Deep Neural Network (DNN) models expanding to risk-sensitive and safety-critical domains, many methods have been proposed to explain the decisions of these models. Recent years have also seen concerted efforts that have shown how such explanations can be distorted (attacked) by minor input perturbations. While there have been many surveys that review explainability methods themselves, there has been no effort hitherto to assimilate the different m…
For sensitive problems, such as medical imaging or fraud detection, Neural Network (NN) adoption has been slow due to concerns about their reliability, leading to a number of algorithms for explaining their decisions. NNs have also been found vulnerable to a class of imperceptible attacks, called adversarial examples, which arbitrarily alter the output of the network. Here we demonstrate both that these attacks can invalidate prior attempts to explain the decisions of NNs, and that with very robust networks, the attacks themselves may be leveraged as explanations with greater fidelity to the m…
Fujitsu Laboratories of America（美国富士通实验室）；University of California, Irvine（加州大学欧文分校）
Financial decisions impact our lives, and thus everyone from the regulator to the consumer is interested in fair, sound, and explainable decisions. There is increasing competitive desire and regulatory incentive to deploy AI mindfully within financial services. An important mechanism towards that end is to explain AI decisions to various stakeholders. State-of-the-art explainable AI systems mostly serve AI engineers and offer little to no value to business decision makers, customers, and other stakeholders. Towards addressing this gap, in this work we consider the scenario of explaining loan d…
Centrum Wiskunde & Informatica（数学与计算机科学中心）；University of Trieste（的里雅斯特大学）
Counterfactual explanations (CEs) are a powerful means for understanding how decisions made by algorithms can be changed. Researchers have proposed a number of desiderata that CEs should meet to be practically useful, such as requiring minimal effort to enact, or complying with causal models. We consider a further aspect to improve the usability of CEs: robustness to adverse perturbations, which may naturally happen due to unfortunate circumstances. Since CEs typically prescribe a sparse form of intervention (i.e., only a subset of the features should be changed), we study the effect of addres…
Performance of diverse evaluation metrics in NLP-based assessment and text generation of consumer complaints
School of Mathematical and Statistical Sciences, Western University（数学与统计科学学院，西部大学）；Department of Population Health Science and Policy, Icahn School of Medicine at Mount Sinai（流行病学与政策部门，辛辛那提医学院）；Institute for Health Care Delivery Science, Icahn School of Medicine at Mount Sinai（健康护理科学研究所，辛辛那提医学院）
MPI-SWS（马克斯·普朗克软件系统研究所）；University of Maryland（马里兰大学）；Bosch Center for Artificial Intelligence（博世人工智能中心）
As deep neural networks (DNNs) get adopted in an ever-increasing number of applications, explainability has emerged as a crucial desideratum for these models. In many real-world tasks, one of the principal reasons for requiring explainability is to in turn assess prediction robustness, where predictions (i.e., class labels) that do not conform to their respective explanations (e.g., presence or absence of a concept in the input) are deemed to be unreliable. However, most, if not all, prior methods for checking explanation-conformity (e.g., LIME, TCAV, saliency maps) require significant manual …
DiCE-Extended: A Robust Approach to Counterfactual Explanations in Machine Learning
Faculty of Graduate Education Institute, Department of Artificial Intelligence (Interdisciplinary), Bahçeşehir University, Turkey（研究生教育学院人工智能系（跨学科）巴塞希尔大学，土耳其）；School of Computer Science, University College Dublin, Ireland（计算机科学学院都柏林大学学院，爱尔兰）；Faculty of Engineering and Natural Sciences, Department of Mathematics, Bahçeşehir University, Turkey（工程与自然科学学院数学系巴塞希尔大学，土耳其）
Explanation-Guided Adversarial Training for Robust and Interpretable Models
Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））；Fuzhou University（福州大学）；NLPR, MAIS, Institute of Automation, Chinese Academy of Sciences（神经网络与模式识别实验室、机器学习与智能系统研究所、自动化研究所，中国科学院）
A Theoretical Explanation of Activation Sparsity through Flat Minima and Adversarial Robustness
State Key Laboratory for Novel Software Technology（计算机软件新技术全国重点实验室）；Nanjing University（南京大学）；School of Computer Science and Engineering（计算机科学与工程学院）；Southeast University（东南大学）
A unified framework for evaluating the robustness of machine-learning interpretability for prospect risking
Center for Energy and Geo Processing (CeGP)（能源与地球处理中心）；Omni Lab for Intelligent Visual Engineering and Science (OLIVES)（智能视觉工程与科学实验室）；School of Electrical and Computer Engineering（电气与计算机工程学院）；Georgia Institute of Technology（佐治亚理工学院）
