Post-hoc analysis is a popular category in eXplainable artificial intelligence (XAI) study. In particular, methods that generate heatmaps have been used to explain the deep neural network (DNN), a black-box model. Heatmaps can be appealing due to the intuitive and visual ways to understand them but assessing their qualities might not be straightforward. Different ways to assess heatmaps' quality have their own merits and shortcomings. This paper introduces a synthetic dataset that can be generated adhoc along with the ground-truth heatmaps for more objective quantitative assessment. Each sample data is an image of a cell with easily recognized features that are distinguished from localization ground-truth mask, hence facilitating a more transparent assessment of different XAI methods. Comparison and recommendations are made, shortcomings are clarified along with suggestions for future research directions to handle the finer details of select post-hoc analysis methods. Furthermore, mabCAM is introduced as the heatmap generation method compatible with our ground-truth heatmaps. The framework is easily generalizable and uses only standard deep learning components.
Graph Neural Networks (GNNs) are deep learning models that take graph data as inputs, and they are applied to various tasks such as traffic prediction and molecular property prediction. However, owing to the complexity of the GNNs, it has been difficult to analyze which parts of inputs affect the GNN model's outputs. In this study, we extend explainability methods for Convolutional Neural Networks (CNNs), such as Local Interpretable Model-Agnostic Explanations (LIME), Gradient-Based Saliency Maps, and Gradient-Weighted Class Activation Mapping (Grad-CAM) to GNNs, and predict which edges in the…
In this paper, we address the problem of quantifying reliability of computational saliency for videos, which can be used to improve saliency-based video processing and enable more reliable performance and risk assessment of such processing. Our approach is twofold. First, we explore spatial correlations in both saliency map and eye-fixation map. Then, we learn spatiotemporal correlations that define a reliable saliency map. We first study spatiotemporal eye-fixation data from a public dataset and investigate a common feature in human visual attention, which dictates correlation in saliency bet…
University of Washington（华盛顿大学）；University of Maryland（马里兰大学）；New York University（纽约大学）
Conventional saliency maps highlight input features to which neural network predictions are highly sensitive. We take a different approach to saliency, in which we identify and analyze the network parameters, rather than inputs, which are responsible for erroneous decisions. We find that samples which cause similar parameters to malfunction are semantically similar. We also show that pruning the most salient parameters for a wrongly classified sample often improves model behavior. Furthermore, fine-tuning a small number of the most salient parameters on a single sample results in error correct…
Traffic accident anticipation is a vital function of Automated Driving Systems (ADSs) for providing a safety-guaranteed driving experience. An accident anticipation model aims to predict accidents promptly and accurately before they occur. Existing Artificial Intelligence (AI) models of accident anticipation lack a human-interpretable explanation of their decision-making. Although these models perform well, they remain a black-box to the ADS users, thus difficult to get their trust. To this end, this paper presents a Gated Recurrent Unit (GRU) network that learns spatio-temporal relational fea…
Learning Temporal Saliency for Time Series Forecasting with Cross-Scale Attention
Department of Software Engineering, Faculty of Computer and Information Sciences, Sakarya University（软件工程系，计算机与信息科学学院，萨卡里亚大学）；Department of Computer and Information Science (IDA), REAL, AIICS, Linkoping University（计算机与信息科学系（IDA），REAL，AIICS，林奈大学）
SCOPE: Saliency-Coverage Oriented Token Pruning for Efficient Multimodel LLMs
University of Electronic Science and Technology of China（电子科技大学）；Shenzhen Institute for Advanced Study（深圳先进研究 institute）；CFAR, Agency for Science, Technology and Research (A*STAR)（科技研究局（A*STAR）认知与人工智能研究中心）；IHPC, Agency for Science, Technology and Research (A*STAR)（科技研究局（A*STAR）人工智能中心）
Understanding the reasons behind the predictions made by deep neural networks is critical for gaining human trust in many important applications, which is reflected in the increasing demand for explainability in AI (XAI) in recent years. Saliency-based feature attribution methods, which highlight important parts of images that contribute to decisions by classifiers, are often used as XAI methods, especially in the field of computer vision. In order to compare various saliency-based XAI methods quantitatively, several approaches for automated evaluation schemes have been proposed; however, ther…
Cranfield University（克兰菲尔德大学）；University of Warwick（华威大学）；Alan Turing Institute（艾伦·图灵研究所）
Achieving transparency in black-box deep learning algorithms is still an open challenge. High dimensional features and decisions given by deep neural networks (NN) require new algorithms and methods to expose its mechanisms. Current state-of-the-art NN interpretation methods (e.g. Saliency maps, DeepLIFT, LIME, etc.) focus more on the direct relationship between NN outputs and inputs rather than the NN structure and operations itself. In current deep NN operations, there is uncertainty over the exact role played by neurons with fixed activation functions. In this paper, we achieve partially ex…
The emerging field of Explainable Artificial Intelligence focuses on researching methods of explaining the decision making processes of complex machine learning models. In the field of explainability for Computer Vision, explanations are provided as saliency maps, which visualize the importance of individual pixels of the input w.r.t. the model's prediction. In this work we focus on a perturbation-based, model-agnostic explainability method called RISE, elaborate on observed shortcomings of its grid-based approach and propose two modifications: replacement of square occlusions with convex poly…
Weizmann Institute of Science（魏茨曼科学研究学院）；Jether Energy Research（Jether能源研究院）
Saliency methods are a popular approach for model debugging and explainability. However, in the absence of ground-truth data for what the correct maps should be, evaluating and comparing different approaches remains a long-standing challenge. The sanity checks methodology of Adebayo et al [Neurips 2018] has sought to address this challenge. They argue that some popular saliency methods should not be used for explainability purposes since the maps they produce are not sensitive to the underlying model that is to be explained. Through a causal re-framing of their objective, we argue that their e…
UCSF（加州大学旧金山分校）；SFVAMC（旧金山退伍军人事务医疗中心）；UC Berkeley（加州大学伯克利分校）；Helen Diller Family Comprehensive Cancer Center, UCSF（加州大学旧金山分校海伦·迪勒家族综合癌症中心）；Bakar Computational Health Sciences Institute, UCSF（加州大学旧金山分校巴卡尔计算健康科学研究所）
Interpretability methods for image classification assess model trustworthiness by attempting to expose whether the model is systematically biased or attending to the same cues as a human would. Saliency methods for feature attribution dominate the interpretability literature, but these methods do not address semantic concepts such as the textures, colors, or genders of objects within an image. Our proposed Robust Concept Activation Vectors (RCAV) quantifies the effects of semantic concepts on individual model predictions and on model behavior as a whole. RCAV calculates a concept gradient and …
Saliency strikes back: How filtering out high frequencies improves white-box explanations
Carney Institute for Brain Science, Brown University（布朗大学卡尼脑科学研究所）；CerCo, CNRS（法国国家科学研究中心CerCo研究所）；SNCF（法国国家铁路公司）；Institute of Mathematics of Toulouse, University Paul Sabatier（保罗萨巴蒂埃大学图卢兹数学研究所）
CYENS Centre of Excellence（CYENS卓越中心）；University of Twente（特文特大学）
The widespread use of black-box AI models has raised the need for algorithms and methods that explain the decisions made by these models. In recent years, the AI research community is increasingly interested in models' explainability since black-box models take over more and more complicated and challenging tasks. Explainability becomes critical considering the dominance of deep learning techniques for a wide range of applications, including but not limited to computer vision. In the direction of understanding the inference process of deep learning models, many methods that provide human compr…
UCSF（加州大学旧金山分校）；SFVAMC（旧金山退伍军人事务医疗中心）；Helen Diller Family Comprehensive Cancer Center（海伦·迪勒家庭综合癌症中心）；Institute for Neurodegenerative Diseases（神经退行性疾病研究所）；Bakar Computational Health Sciences Institute（巴卡尔计算健康科学研究所）
In high-stakes applications of machine learning models, interpretability methods provide guarantees that models are right for the right reasons. In medical imaging, saliency maps have become the standard tool for determining whether a neural model has learned relevant robust features, rather than artefactual noise. However, saliency maps are limited to local model explanation because they interpret predictions on an image-by-image basis. We propose aggregating saliency globally, using semantic segmentation masks, to provide quantitative measures of model bias across a dataset. To evaluate glob…
Huawei Ireland Research Center（华为爱尔兰研究中心）；Boston University（波士顿大学）；Istituto Italiano di Tecnologia（意大利理工学院）；Naver Labs Europe（欧洲Naver实验室）；Microsoft（微软）；Adobe Research（Adobe研究院）；Università di Verona（维罗纳大学）
Conventionally, AI models are thought to trade off explainability for lower accuracy. We develop a training strategy that not only leads to a more explainable AI system for object classification, but as a consequence, suffers no perceptible accuracy degradation. Explanations are defined as regions of visual evidence upon which a deep classification network makes a decision. This is represented in the form of a saliency map conveying how much each pixel contributed to the network's decision. Our training strategy enforces a periodic saliency-based feedback to encourage the model to focus on the…
IIT Kharagpur（印度理工学院卡拉格普尔分校）；University of Poitiers（普瓦捷大学）；IIT Jodhpur（印度理工学院焦特布尔分校）
Deep neural networks have shown their profound impact on achieving human level performance in visual saliency prediction. However, it is still unclear how they learn the task and what it means in terms of understanding human visual system. In this work, we develop a technique to derive explainable saliency models from their corresponding deep neural architecture based saliency models by applying human perception theories and the conventional concepts of saliency. This technique helps us understand the learning pattern of the deep network at its intermediate layers through their activation maps…
Signature Activation: A Sparse Signal View for Holistic Saliency
Harvard University（哈佛大学）；Broad Institute of Massachusetts Institute of Technology and Harvard（麻省理工学院与哈佛大学博德研究所）；Massachusetts General Hospital（麻省总医院）；Harvard Medical School（哈佛医学院）
AI4Health Division, Faculty VI Carl von Ossietzky Universität Oldenburg（AI4Health部门，第六学院卡尔·冯·奥西特齐克大学）；Semantic Information Systems Group Osnabrück University（语义信息系统小组奥尔纳布里克大学）
Medical Slice Transformer: Improved Diagnosis and Explainability on 3D Medical Images with DINOv2
University Hospital Aachen（亚琛大学医院）；Technical University Dresden（德累斯顿工业大学）；University Hospital Dresden（德累斯顿大学医院）；University Hospital Heidelberg（海德堡大学医院）
Caption-Driven Explainability: Probing CNNs for Bias via CLIP
Northwestern University, Evanston, IL, USA（西北大学）；University of California, Berkeley, CA, USA（加州大学伯克利分校）；Eastern Switzerland University of Applied Sciences, Rapperswil, SG, CH（东瑞士应用科学大学）
IBM Research Zurich（苏黎世IBM研究公司）；EPFL Lausanne（洛桑联邦理工学院）；ETH Zurich（苏黎世联邦理工学院）；Fondazione Pascale（帕斯卡尔基金会）；University of Bern（伯尔尼大学）；Uppsala University（乌普萨拉大学）
Explainability of deep learning methods is imperative to facilitate their clinical adoption in digital pathology. However, popular deep learning methods and explainability techniques (explainers) based on pixel-wise processing disregard biological entities' notion, thus complicating comprehension by pathologists. In this work, we address this by adopting biological entity-based graph processing and graph explainers enabling explanations accessible to pathologists. In this context, a major challenge becomes to discern meaningful explainers, particularly in a standardized and quantifiable fashio…
An Experimental Investigation into the Evaluation of Explainability Methods
University of Mons（蒙斯大学）；UCLouvain（鲁汶大学）；Cliniques Universitaires Saint-Luc（圣卢克大学诊所）；IREC（IREC研究所）；University of Namur（那慕尔大学）；Interuniversity Institute for Bioinformatics in Brussels（布鲁塞尔跨大学生物信息学研究所）；Université Libre de Bruxelles（布鲁塞尔自由大学）；Multitel（Multitel研究中心）
Classification Metrics for Image Explanations: Towards Building Reliable XAI-Evaluations
Fraunhofer Institute for Manufacturing Engineering and Automation IPA（弗劳恩霍夫制造工程与自动化研究所）；Institute of Industrial Manufacturing and Management IFF, University of Stuttgart（斯图加特大学工业制造与管理研究所）
Skin cancer is considered to be the most common human malignancy. Around 5 million new cases of skin cancer are recorded in the United States annually. Early identification and evaluation of skin lesions is of great clinical significance, but the disproportionate dermatologist-patient ratio poses significant problem in most developing nations. Therefore a deep learning based architecture, known as SkiNet, is proposed with an objective to provide faster screening solution and assistance to newly trained physicians in the clinical diagnosis process. The main motive behind Skinet's design and dev…
UbiQVision: Quantifying Uncertainty in XAI for Image Recognition
Center for Artificial Intelligence in Public Health Research (ZKI-PH), Robert Koch Institute（公共健康人工智能研究中心（ZKI-PH），罗伯特·科赫研究所）；Department of Mathematics and Computer Science, Freie Universität Berlin（数学与计算机科学系，柏林自由大学）
On the explainable properties of 1-Lipschitz Neural Networks: An Optimal Transport Perspective
Université Paul-Sabatier（保罗萨巴蒂尔大学）；Institut de Recherche Technologique Saint-Exupéry（圣埃克苏佩里技术研究院）；Carney Institute for Brain Science,Brown University（布朗大学卡尼脑科学研究所）；Innovation & Research Division, SNCF（法国国家铁路公司创新与研究部）
University of Notre Dame（圣母大学）；Naval Surface Warfare Center – Crane Division（海军水面作战中心克兰分部）
The performance of convolutional neural networks has continued to improve over the last decade. At the same time, as model complexity grows, it becomes increasingly more difficult to explain model decisions. Such explanations may be of critical importance for reliable operation of human-machine pairing setups, or for model selection when the "best" model among many equally-accurate models must be established. Saliency maps represent one popular way of explaining model decisions by highlighting image regions models deem important when making a prediction. However, examining salience maps at sca…
Bridging visual saliency and large language models for explainable deep learning in medical imaging
Department of Mathematics and Computer science（数学与计算机科学系）；Institute of Cognitive Science（认知科学研究所）；Rhodes University（罗德斯大学）；National Institute for Theoretical and Computational Sciences (NITheCS)（理论与计算科学国家研究所）
Explaining Digital Pathology Models via Clustering Activations
Faculty of Informatics, Masaryk University, Brno, Czech Republic（马萨里克大学信息学院，布拉格，捷克共和国）；Institute of Computer Science, Masaryk University, Brno, Czech Republic（马萨里克大学计算机科学研究所，布拉格，捷克共和国）；Masaryk Memorial Cancer Institute, Brno, Czech Republic（马萨里克纪念癌症研究所，布拉格，捷克共和国）
Explainability is an important factor to drive user trust in the use of neural networks for tasks with material impact. However, most of the work done in this area focuses on image analysis and does not take into account 3D data. We extend the saliency methods that have been shown to work on image data to deal with 3D data. We analyse the features in point clouds and voxel spaces and show that edges and corners in 3D data are deemed as important features while planar surfaces are deemed less important. The approach is model-agnostic and can provide useful information about learnt features. Dri…
