Attribution methods have emerged as a popular approach to interpreting model predictions based on the relevance of input features. Although the feature importance ranking can provide insights of how models arrive at a prediction from a raw input, they do not give a clear-cut definition of the key features models use for the prediction. In this paper, we present a new method, called WheaCha, for explaining the predictions of code models. Although WheaCha employs the same mechanism of tracing model predictions back to the input features, it differs from all existing attribution methods in crucial ways. Specifically, WheaCha divides an input program into "wheat" (i.e., the defining features that are the reason for which models predict the label that they predict) and the rest "chaff" for any prediction of a learned code model. We realize WheaCha in a tool, HuoYan, and use it to explain four prominent code models: code2vec, seq-GNN, GGNN, and CodeBERT. Results show (1) HuoYan is efficient - taking on average under twenty seconds to compute the wheat for an input program in an end-to-end fashion (i.e., including model prediction time); (2) the wheat that all models use to predict input programs is made of simple syntactic or even lexical properties (i.e., identifier names); (3) Based on wheat, we present a novel approach to explaining the predictions of code models through the lens of training data.
Towards Few-shot Self-explaining Graph Neural Networks
University of Science and Technology of China（中国科学技术大学）；Institute of Artificial Intelligence Hefei Comprehensive National Science Center（合肥综合性国家科学中心人工智能研究院）
The George Washington University（乔治华盛顿大学）；University of North Texas（北德克萨斯大学）
Graph neural networks (GNNs) have been utilized to create multi-layer graph models for a number of cybersecurity applications from fraud detection to software vulnerability analysis. Unfortunately, like traditional neural networks, GNNs also suffer from a lack of transparency, that is, it is challenging to interpret the model predictions. Prior works focused on specific factor explanations for a GNN model. In this work, we have designed and implemented Illuminati, a comprehensive and accurate explanation framework for cybersecurity applications using GNN models. Given a graph and a pre-trained…
School of Computer Sciences（计算机科学学院）；National Institute of Science Education and Research（国家科学教育与研究 institute）；An OCC of Homi Bhabha National Institute, India（印度霍米·巴哈瓦国家研究所的一个分部）
Relevant Walk Search for Explaining Graph Neural Networks
BIFOLD -- Berlin Institute for the Foundations of Learning（柏林学习与数据基础研究所）；Google Research, Brain team, Berlin（谷歌研究，柏林脑团队）；Department of Artificial Intelligence, Korea University, Seoul 136-713, Korea（人工智能系，韩国大学，首尔136-713，韩国）；RIKEN Center for AIP, Japan（日本AIP研究中心）
Toward Explaining Large Language Models in Software Engineering Tasks
University of Molise \& Politecnico di Torino Termoli Italy；University of Molise Italy；University of Molise \& Politecnico di Torino；University of Molise
Allen Institute for Artificial Intelligence（艾伦人工智能研究所）；University of California, Irvine（加州大学欧文分校）
Neural NLP models are increasingly accurate but are imperfect and opaque---they break in counterintuitive ways and leave end users puzzled at their behavior. Model interpretation methods ameliorate this opacity by providing explanations for specific model predictions. Unfortunately, existing interpretation codebases make it difficult to apply these methods to new models and tasks, which hinders adoption for practitioners and burdens interpretability researchers. We introduce AllenNLP Interpret, a flexible framework for interpreting NLP models. The toolkit provides interpretation primitives (e.…
Explaining AI Without Code: A User Study on Explainable AI
Department of Computer Science, University of Chile（智利大学计算机科学系）；CENIA – National Center for Artificial Intelligence（人工智能国家研究中心）；Universidad Técnica Federico Santa María（弗朗西斯科·桑塔玛利亚技术大学）；IMFD - Millennium Institute for Foundational Research on Data（数据基础研究千年研究所）
Recent work has shown great promise in explaining neural network behavior. In particular, feature attribution methods explain which features were most important to a model's prediction on a given input. However, for many tasks, simply knowing which features were important to a model's prediction may not provide enough insight to understand model behavior. The interactions between features within the model may better help us understand not only the model, but also why certain features are more important than others. In this work, we present Integrated Hessians, an extension of Integrated Gradie…
Beyond Fidelity: Explaining Vulnerability Localization of Learning-based Detectors
Peking University（北京大学）；University of Alberta（阿尔伯塔大学）；Huazhong University of Science and Technology（华中科技大学）；University of Queensland（昆士兰大学）；University of New South Wales（新南威尔士大学）
Technical University of Munich（慕尼黑工业大学）；Kyung Hee University（庆熙大学）；Johns Hopkins University（约翰斯·霍普金斯大学）
Neural networks have demonstrated remarkable performance in classification and regression tasks on chest X-rays. In order to establish trust in the clinical routine, the networks' prediction mechanism needs to be interpretable. One principal approach to interpretation is feature attribution. Feature attribution methods identify the importance of input features for the output prediction. Building on Information Bottleneck Attribution (IBA) method, for each prediction we identify the chest X-ray regions that have high mutual information with the network's output. Original IBA identifies input re…
Slovak University of Technology（斯洛伐克理工大学）；Johannes Kepler University（约翰内斯·开普勒大学）
In many practical applications, deep neural networks have been typically deployed to operate as a black box predictor. Despite the high amount of work on interpretability and high demand on the reliability of these systems, they typically still have to include a human actor in the loop, to validate the decisions and handle unpredictable failures and unexpected corner cases. This is true in particular for failure-critical application domains, such as medical diagnosis. We present a novel approach to explain and support an interpretation of the decision-making process to a human expert operating…
