解释图神经网络（GNN）的节点预测通常可归结为寻找能保留预测结果的图子结构。寻找这类结构通常需要沿GNN进行反向传播，这使得GNN的复杂度（如层数）与解释成本直接绑定。这自然引出一个问题：我们能否通过解释更简单的代理GNN来打破这种绑定？为回答该问题，我们提出Distill n' Explain（DnX）方法。首先，DnX通过知识蒸馏学习一个代理GNN；随后，DnX通过求解一个简单的凸规划提取节点级或边级的解释。我们还提出了FastDnX，这是DnX的更快版本，利用了代理模型的线性分解特性。实验表明，DnX和FastDnX的性能通常优于当前最先进的GNN解释器，同时速度快数个数量级。此外，我们还通过理论结果支撑了实证发现，该理论将代理模型的质量（即蒸馏误差）与解释的忠实度关联起来。
英文摘要：
Explaining node predictions in graph neural networks (GNNs) often boils down to finding graph substructures that preserve predictions. Finding these structures usually implies back-propagating through the GNN, bonding the complexity (e.g., number of layers) of the GNN to the cost of explaining it. This naturally begs the question: Can we break this bond by explaining a simpler surrogate GNN? To answer the question, we propose Distill n' Explain (DnX). First, DnX learns a surrogate GNN via knowledge distillation. Then, DnX extracts node or edge-level explanations by solving a simple convex program. We also propose FastDnX, a faster version of DnX that leverages the linear decomposition of our surrogate model. Experiments show that DnX and FastDnX often outperform state-of-the-art GNN explainers while being orders of magnitude faster. Additionally, we support our empirical findings with theoretical results linking the quality of the surrogate model (i.e., distillation error) to the faithfulness of explanations.
Graph Neural Networks (GNNs) have become increasingly ubiquitous in numerous applications and systems, necessitating explanations of their predictions, especially when making critical decisions. However, explaining GNNs is challenging due to the complexity of graph data and model execution. Despite additional computational costs, post-hoc explanation approaches have been widely adopted due to the generality of their architectures. Intrinsically interpretable models provide instant explanations but are usually model-specific, which can only explain particular GNNs. Therefore, we propose a novel…
School of Computer Sciences（计算机科学学院）；National Institute of Science Education and Research（国家科学教育与研究 institute）；An OCC of Homi Bhabha National Institute, India（印度霍米·巴哈瓦国家研究所的一个分部）
We propose a novel method to explain trained deep neural networks (DNNs), by distilling them into surrogate models using unsupervised clustering. Our method can be applied flexibly to any subset of layers of a DNN architecture and can incorporate low-level and high-level information. On image datasets given pre-trained DNNs, we demonstrate the strength of our method in finding similar training samples, and shedding light on the concepts the DNNs base their decisions on. Via user studies, we show that our model can improve the user trust in model's prediction.
Tsinghua University（清华大学）；National University of Singapore（新加坡国立大学）
Deep neural networks (DNNs) demonstrate great success in classification tasks. However, they act as black boxes and we don't know how they make decisions in a particular classification task. To this end, we propose to distill the knowledge from a DNN into a fuzzy inference system (FIS), which is Takagi-Sugeno-Kang (TSK)-type in this paper. The model has the capability to express the knowledge acquired by a DNN based on fuzzy rules, thus explaining a particular decision much easier. Knowledge distillation (KD) is applied to create a TSK-type FIS that generalizes better than one directly from th…
Deep neural networks have proved to be a very effective way to perform classification tasks. They excel when the input data is high dimensional, the relationship between the input and the output is complicated, and the number of labeled training examples is large. But it is hard to explain why a learned network makes a particular classification decision on a particular test case. This is due to their reliance on distributed hierarchical representations. If we could take the knowledge acquired by the neural net and express the same knowledge in a model that relies on hierarchical decisions inst…
Texas State University（德克萨斯州立大学）；Illinois Institute of Technology（伊利诺伊理工学院）
In recent years the ubiquitous deployment of AI has posed great concerns in regards to algorithmic bias, discrimination, and fairness. Compared to traditional forms of bias or discrimination caused by humans, algorithmic bias generated by AI is more abstract and unintuitive therefore more difficult to explain and mitigate. A clear gap exists in the current literature on evaluating and mitigating bias in pruned neural networks. In this work, we strive to tackle the challenging issues of evaluating, mitigating, and explaining induced bias in pruned neural networks. Our paper makes three contribu…
University of Alberta（阿尔伯塔大学）；Alberta Machine Intelligence Institute (Amii)（阿尔伯塔机器智能研究院）
One major barrier to applications of deep Reinforcement Learning (RL) both inside and outside of games is the lack of explainability. In this paper, we describe a lightweight and effective method to derive explanations for deep RL agents, which we evaluate in the Atari domain. Our method relies on a transformation of the pixel-based input of the RL agent to an interpretable, percept-like input representation. We then train a surrogate model, which is itself interpretable, to replicate the behavior of the target, deep RL agent. Our experiments demonstrate that we can learn an effective surrogat…
Stony Brook University（石溪大学）；Sapienza University of Rome（罗马第一大学）；The Xi’an Jiaotong-Liverpool University（西交利物浦大学）；Rutgers University（罗格斯大学）；South China Normal University（华南师范大学）
Recently, graph neural networks (GNNs) have been widely used to develop successful recommender systems. Although powerful, it is very difficult for a GNN-based recommender system to attach tangible explanations of why a specific item ends up in the list of suggestions for a given user. Indeed, explaining GNN-based recommendations is unique, and existing GNN explanation methods are inappropriate for two reasons. First, traditional GNN explanation methods are designed for node, edge, or graph classification tasks rather than ranking, as in recommender systems. Second, standard machine learning e…
Information-Geometric Inverse Distillation for Enhancing Adversarial Transferability
College of Computer Science, Sichuan University（四川大学计算机学院）；Sichuan Newstrong UHD Video Technology Company Ltd.（四川新创视超高清视频科技有限公司）；Institute of Optics and Electronics, Chinese Academy of Sciences（中国科学院光电技术研究所）
Sberbank AI Lab（俄罗斯联邦储蓄银行人工智能实验室）；Samsung AI Center in Moscow（三星莫斯科人工智能中心）；Stern School of Business, NYU（纽约大学斯特恩商学院）
We introduce a novel approach to feed-forward neural network interpretation based on partitioning the space of sequences of neuron activations. In line with this approach, we propose a model-specific interpretation method, called YASENN. Our method inherits many advantages of model-agnostic distillation, such as an ability to focus on the particular input region and to express an explanation in terms of features different from those observed by a neural network. Moreover, examination of distillation error makes the method applicable to the problems with low tolerance to interpretation mistakes…
Model Mimic Attack: Knowledge Distillation for Provably Transferable Adversarial Examples
Research Center for Trusted Artificial Intelligence（可信人工智能研究中心）；Ivannikov Institute for System Programming of the Russian Academy of Sciences（俄罗斯科学院伊万尼科夫系统编程研究所）；Moscow Institute of Physics and Technology (National Research University)（莫斯科物理技术学院（国立研究型大学））；AIRI（AIRI（人工智能研究所））
University of Amsterdam（阿姆斯特丹大学）；Ahold Delhaize（皇家阿霍德德尔海兹集团）
Understanding how "black-box" models arrive at their predictions has sparked significant interest from both within and outside the AI community. Our work focuses on doing this by generating local explanations about individual predictions for tree-based ensembles, specifically Gradient Boosting Decision Trees (GBDTs). Given a correctly predicted instance in the training set, we wish to generate a counterfactual explanation for this instance, that is, the minimal perturbation of this instance such that the prediction flips to the opposite class. Most existing methods for counterfactual explanati…
With the ever-increasing use of complex machine learning models in critical applications within the finance domain, explaining the decisions of the model has become a necessity. With applications spanning from credit scoring to credit marketing, the impact of these models is undeniable. Among the multiple ways in which one can explain the decisions of these complicated models, local post hoc model agnostic explanations have gained massive adoption. These methods allow one to explain each prediction independent of the modelling technique that was used while training. As explanations, they eithe…
In Machine Learning as a Service, a provider trains a deep neural network and gives many users access. The hosted (source) model is susceptible to model stealing attacks, where an adversary derives a surrogate model from API access to the source model. For post hoc detection of such attacks, the provider needs a robust method to determine whether a suspect model is a surrogate of their model. We propose a fingerprinting method for deep neural network classifiers that extracts a set of inputs from the source model so that only surrogates agree with the source model on the classification of such…
Jheronimus Academy of Data Science（杰罗尼姆斯数据科学学院）；University of Amsterdam（阿姆斯特丹大学）；Tilburg University（蒂尔堡大学）；Eindhoven University of Technology（埃因霍温理工大学）
Constructing accurate model-agnostic explanations for opaque machine learning models remains a challenging task. Classification models for high-dimensional data, like images, are often inherently complex. To reduce this complexity, individual predictions may be explained locally, either in terms of a simpler local surrogate model or by communicating how the predictions contrast with those of another class. However, existing approaches still fall short in the following ways: a) they measure locality using a (Euclidean) metric that is not meaningful for non-linear high-dimensional data; or b) th…
University of California, Davis（加州大学戴维斯分校）；Zhejiang University（浙江大学）；University of California（加州大学）
Deep Recurrent Neural Networks (RNN) is increasingly used in decision-making with temporal sequences. However, understanding how RNN models produce final predictions remains a major challenge. Existing work on interpreting RNN models for sequence predictions often focuses on explaining predictions for individual data instances (e.g., patients or students). Because state-of-the-art predictive models are formed with millions of parameters optimized over millions of instances, explaining predictions for single data instances can easily miss a bigger picture. Besides, many outperforming RNN models…
Explaining Deep Network Classification of Matrices: A Case Study on Monotonicity
Institute of Mathematics and Statistics, Federal University of Rio Grande do Sul（数学与统计学学院，里约格朗德杜斯尔大学）；Division of Mathematics and Physics, UKK, Mälardalen University（数学与物理系，马尔达伦大学）
Adversarial Vulnerabilities in Neural Operator Digital Twins: Gradient-Free Attacks on Nuclear Thermal-Hydraulic Surrogates
Department of Nuclear, Plasma & Radiological Engineering, University of Illinois Urbana-Champaign（核物理与辐射工程系，伊利诺伊大学厄巴纳-香槟分校）；Indian Institute of Technology Delhi（印度理工学院德里）；National Center for Supercomputing Applications, University of Illinois Urbana-Champaign（国家超级计算应用中心，伊利诺伊大学厄巴纳-香槟分校）
Tau-BNO: Brain Neural Operator for Tau Transport Model
Department of Radiology and Biomedical Imaging, University of California San Francisco, USA（加州大学旧金山分校放射学与生物医学成像系）；College of Computer Science and Engineering, Northeastern University, China（东北大学计算机科学与工程学院）；Department of Computer Science, Hangzhou Normal University, China（杭州师范大学计算机科学系）；Department of Computer Science, Wake Forest University, USA（威克森林大学计算机科学系）
