For reliability, it is important that the predictions made by machine learning methods are interpretable by human. In general, deep neural networks (DNNs) can provide accurate predictions, although it is difficult to interpret why such predictions are obtained by DNNs. On the other hand, interpretation of linear models is easy, although their predictive performance would be low since real-world data is often intrinsically non-linear. To combine both the benefits of the high predictive performance of DNNs and high interpretability of linear models into a single model, we propose neural generators of sparse local linear models (NGSLLs). The sparse local linear models have high flexibility as they can approximate non-linear functions. The NGSLL generates sparse linear weights for each sample using DNNs that take original representations of each sample (e.g., word sequence) and their simplified representations (e.g., bag-of-words) as input. By extracting features from the original representations, the weights can contain rich information to achieve high predictive performance. Additionally, the prediction is interpretable because it is obtained by the inner product between the simplified representations and the sparse weights, where only a small number of weights are selected by our gate module in the NGSLL. In experiments with real-world datasets, we demonstrate the effectiveness of the NGSLL quantitatively and qualitatively by evaluating prediction performance and visualizing generated weights on image and text classification tasks.
Learning Pseudorandom Numbers with Transformers: Permuted Congruential Generators, Curricula, and Interpretability
Department of Physics, University of Maryland, College Park, USA（大学物理系，马里兰大学，College Park, USA）；Meta FAIR Joint Quantum Institute, University of Maryland（Meta FAIR联合量子研究所，马里兰大学）
A new understanding of adversarial examples and adversarial robustness is proposed by decoupling the data generator and the label generator (which we call the teacher). In our framework, adversarial robustness is a conditional concept---the student model is not absolutely robust, but robust with respect to the teacher. Based on the new understanding, we claim that adversarial examples exist because the student cannot obtain sufficient information of the teacher from the training data. Various ways of achieving robustness is compared. Theoretical and numerical evidence shows that to efficiently…
Ongoing efforts to understand deep neural networks (DNN) have provided many insights, but DNNs remain incompletely understood. Improving DNN's interpretability has practical benefits, such as more accountable usage, better algorithm maintenance and improvement. The complexity of dataset structure may contribute to the difficulty in solving interpretability problem arising from DNN's black-box mechanism. Thus, we propose to use pattern theory formulated by Ulf Grenander, in which data can be described as configurations of fundamental objects that allow us to investigate convolutional neural net…
University of Liverpool（利物浦大学）；University of Oxford（牛津大学）；Lancaster University（兰卡斯特大学）；Defence Science and Technology Laboratory (Dstl)（国防科学技术实验室）；Queen’s University Belfast（贝尔法斯特女王大学）
In the past few years, significant progress has been made on deep neural networks (DNNs) in achieving human-level performance on several long-standing tasks. With the broader deployment of DNNs on various applications, the concerns over their safety and trustworthiness have been raised in public, especially after the widely reported fatal incidents involving self-driving cars. Research to address these concerns is particularly active, with a significant number of papers released in the past few years. This survey paper conducts a review of the current research effort into making DNNs safe and …
Recidivism prediction scores are used across the USA to determine sentencing and supervision for hundreds of thousands of inmates. One such generator of recidivism prediction scores is Northpointe's Correctional Offender Management Profiling for Alternative Sanctions (COMPAS) score, used in states like California and Florida, which past research has shown to be biased against black inmates according to certain measures of fairness. To counteract this racial bias, we present an adversarially-trained neural network that predicts recidivism and is trained to remove racial bias. When comparing the…
Mechanistic Interpretability for Neural TSP Solvers
University of Washington, Michael G. Foster School of Business（华盛顿大学，迈克尔·G·福斯特商学院）；University of Washington, Paul G. Allen School of Computer Science & Engineering（华盛顿大学，保罗·G·阿伦计算机科学与工程学院）
KoopGen: Koopman Generator Networks for Representing and Predicting Dynamical Systems with Continuous Spectra
School of Mathematics and Statistics and Ministry of Education Key Lab of Intelligent Networks and Network Security（数学与统计学学院和教育部智能网络与网络安全重点实验室）；School of Mathematics（数学学院）
Towards Accurate and Interpretable Time-series Forecasting: A Polynomial Learning Approach
Center for Intelligent Decision-Making（智能决策中心）；Machine Learning, School of Management, Xi'an Jiaotong University, Xi'an, China（机器学习，管理学院，西安交通大学，中国）；The 39th Research Institute of China Electronics Technology Group Corporation, Xi'an, China（中国电子科技集团第三十九研究所，中国）；Shenzhen Research Institute of Big Data, ShenZhen, China（深圳大数据研究院，中国）
IBM Research（IBM研究院）；KU Leuven（鲁汶大学）；Politecnico di Torino（都灵理工大学）；University of Cambridge（剑桥大学）；Scuola Normale Superiore（高等师范学院）；Università della Svizzera italiana（瑞士意大利语区大学）
While machine learning techniques have been successfully applied in several fields, the black-box nature of the models presents challenges for interpreting and explaining the results. We develop a new framework called Adaptive Explainable Neural Networks (AxNN) for achieving the dual goals of good predictive performance and model interpretability. For predictive performance, we build a structured neural network made up of ensembles of generalized additive model networks and additive index models (through explainable neural networks) using a two-stage process. This can be done using either a bo…
Lawrence Berkeley National Laboratory（劳伦斯伯克利国家实验室）；Toyota Research Institute（丰田研究院）；IMDEA Materials Institute（IMDEA材料研究所）
Machine learning has emerged as a powerful approach in materials discovery. Its major challenge is selecting features that create interpretable representations of materials, useful across multiple prediction tasks. We introduce an end-to-end machine learning model that automatically generates descriptors that capture a complex representation of a material's structure and chemistry. This approach builds on computational topology techniques (namely, persistent homology) and word embeddings from natural language processing. It automatically encapsulates geometric and chemical information directly…
Robust Feature-Level Adversaries are Interpretability Tools
MIT CSAIL（麻省理工学院计算机科学与人工智能实验室）；Boston Children’s Hospital（波士顿儿童医院）；Harvard Medical School（哈佛医学院）；Center for Brains, Minds, and Machines（脑、心智与机器中心）；Harvard College（哈佛学院）；Harvard University（哈佛大学）
Seoul National University（首尔国立大学）；Ajou University（亚洲大学）
How can we effectively find the best structures in tree models? Tree models have been favored over complex black box models in domains where interpretability is crucial for making irreversible decisions. However, searching for a tree structure that gives the best balance between the performance and the interpretability remains a challenging task. In this paper, we propose TART (Transition Matrix Representation with Transposed Convolutions), our novel generalized tree representation for optimal structural search. TART represents a tree model with a series of transposed convolutions that boost t…
Classifiers and generators have long been separated. We break down this separation and showcase that conventional neural network classifiers can generate high-quality images of a large number of categories, being comparable to the state-of-the-art generative models (e.g., DDPMs and GANs). We achieve this by computing the partial derivative of the classification loss function with respect to the input to optimize the input to produce an image. Since it is widely known that directly optimizing the inputs is similar to targeted adversarial attacks incapable of generating human-meaningful images, …
Emory University（埃默里大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Princeton University（普林斯顿大学）；Lehigh University（理海大学）
Interpretable brain network models for disease prediction are of great value for the advancement of neuroscience. GNNs are promising to model complicated network data, but they are prone to overfitting and suffer from poor interpretability, which prevents their usage in decision-critical scenarios like healthcare. To bridge this gap, we propose BrainNNExplainer, an interpretable GNN framework for brain network analysis. It is mainly composed of two jointly learned modules: a backbone prediction model that is specifically designed for brain networks and an explanation generator that highlights …
University of Science and Technology of China（中国科学技术大学）；Tencent America（腾讯美国）
Despite the recent progress in Graph Neural Networks (GNNs), it remains challenging to explain the predictions made by GNNs. Existing explanation methods mainly focus on post-hoc explanations where another explanatory model is employed to provide explanations for a trained GNN. The fact that post-hoc methods fail to reveal the original reasoning process of GNNs raises the need of building GNNs with built-in interpretability. In this work, we propose Prototype Graph Neural Network (ProtGNN), which combines prototype learning with GNNs and provides a new perspective on the explanations of GNNs. …
University of Innsbruck（因斯布鲁克大学）；University of Hamburg（汉堡大学）
The complexity of bipedal locomotion may be attributed to the difficulty in synchronizing joint movements while at the same time achieving high-level objectives such as walking in a particular direction. Artificial central pattern generators (CPGs) can produce synchronized joint movements and have been used in the past for bipedal locomotion. However, most existing CPG-based approaches do not address the problem of high-level control explicitly. We propose a novel hierarchical control mechanism for bipedal locomotion where an optimized CPG network is used for joint control and a neural network…
