Transformer-based models have achieved state-of-the-art performance in various computer vision tasks, including image and video analysis. However, Transformer's complex architecture and black-box nature pose challenges for explainability, a crucial aspect for real-world applications and scientific inquiry. Current Explainable AI (XAI) methods can only provide one-dimensional feature importance, either spatial or temporal explanation, with significant computational complexity. This paper introduces STAA (Spatio-Temporal Attention Attribution), an XAI method for interpreting video Transformer models. Differ from traditional methods that separately apply image XAI techniques for spatial features or segment contribution analysis for temporal aspects, STAA offers both spatial and temporal information simultaneously from attention values in Transformers. The study utilizes the Kinetics-400 dataset, a benchmark collection of 400 human action classes used for action recognition research. We introduce metrics to quantify explanations. We also apply optimization to enhance STAA's raw output. By implementing dynamic thresholding and attention focusing mechanisms, we improve the signal-to-noise ratio in our explanations, resulting in more precise visualizations and better evaluation results. In terms of computational overhead, our method requires less than 3\% of the computational resources of traditional XAI methods, making it suitable for real-time video XAI analysis applications. STAA contributes to the growing field of XAI by offering a method for researchers and practitioners to analyze Transformer models.
This paper aims to explain adversarial attacks in terms of how adversarial perturbations contribute to the attacking task. We estimate attributions of different image regions to the decrease of the attacking cost based on the Shapley value. We define and quantify interactions among adversarial perturbation pixels, and decompose the entire perturbation map into relatively independent perturbation components. The decomposition of the perturbation map shows that adversarially-trained DNNs have more perturbation components in the foreground than normally-trained DNNs. Moreover, compared to the nor…
Institute of Industrial Science, The University of Tokyo（东京大学生产技术研究所）；DUT-RU International School of Information Science and Engineering, Dalian University of Technology（大连理工大学DUT-RU国际信息科学与工程学院）；ETH Zurich（苏黎世联邦理工学院）
The attribution method provides a direction for interpreting opaque neural networks in a visual way by identifying and visualizing the input regions/pixels that dominate the output of a network. Regarding the attribution method for visually explaining video understanding networks, it is challenging because of the unique spatiotemporal dependencies existing in video inputs and the special 3D convolutional or recurrent structures of video understanding networks. However, most existing attribution methods focus on explaining networks taking a single image as input and a few works specifically dev…
Image super-resolution (SR) techniques have been developing rapidly, benefiting from the invention of deep networks and its successive breakthroughs. However, it is acknowledged that deep learning and deep neural networks are difficult to interpret. SR networks inherit this mysterious nature and little works make attempt to understand them. In this paper, we perform attribution analysis of SR networks, which aims at finding the input pixels that strongly influence the SR results. We propose a novel attribution approach called local attribution map (LAM), which inherits the integral gradient me…
From Attribution Maps to Human-Understandable Explanations through Concept Relevance Propagation
Fraunhofer Heinrich-Hertz-Institute（弗劳恩霍夫海因里希-赫兹研究所）；Technische Universität Berlin（柏林工业大学）；BIFOLD – Berlin Institute for the Foundations of Learning and Data（BIFOLD – 柏林学习与数据基础研究所）
The clear transparency of Deep Neural Networks (DNNs) is hampered by complex internal structures and nonlinear transformations along deep hierarchies. In this paper, we propose a new attribution method, Relative Sectional Propagation (RSP), for fully decomposing the output predictions with the characteristics of class-discriminative attributions and clear objectness. We carefully revisit some shortcomings of backpropagation-based attribution methods, which are trade-off relations in decomposing DNNs. We define hostile factor as an element that interferes with finding the attributions of the ta…
University College London（伦敦大学学院）；Comcast Applied AI（康卡斯特应用人工智能研究院）；University of Waterloo（滑铁卢大学）
Large-scale diffusion neural networks represent a substantial milestone in text-to-image generation, but they remain poorly understood, lacking interpretability analyses. In this paper, we perform a text-image attribution analysis on Stable Diffusion, a recently open-sourced model. To produce pixel-level attribution maps, we upscale and aggregate cross-attention word-pixel scores in the denoising subnetwork, naming our method DAAM. We evaluate its correctness by testing its semantic segmentation ability on nouns, as well as its generalized attribution quality on all parts of speech, rated by h…
Korea University（高丽大学）；Tel Aviv University（特拉维夫大学）；KAIST（韩国科学技术院）；Facebook AI Research（脸书人工智能研究院）
As Deep Neural Networks (DNNs) have demonstrated superhuman performance in a variety of fields, there is an increasing interest in understanding the complex internal mechanisms of DNNs. In this paper, we propose Relative Attributing Propagation (RAP), which decomposes the output predictions of DNNs with a new perspective of separating the relevant (positive) and irrelevant (negative) attributions according to the relative influence between the layers. The relevance of each neuron is identified with respect to its degree of contribution, separated into positive and negative, while preserving th…
Clinical applicability of automated decision support systems depends on a robust, well-understood classification interpretation. Artificial neural networks while achieving class-leading scores fall short in this regard. Therefore, numerous approaches have been proposed that map a salient region of an image to a diagnostic classification. Utilizing heuristic methodology, like blurring and noise, they tend to produce diffuse, sometimes misleading results, hindering their general adoption. In this work we overcome these issues by presenting a model agnostic saliency mapping framework tailored to …
We introduce OmniXAI (short for Omni eXplainable AI), an open-source Python library of eXplainable AI (XAI), which offers omni-way explainable AI capabilities and various interpretable machine learning techniques to address the pain points of understanding and interpreting the decisions made by machine learning (ML) in practice. OmniXAI aims to be a one-stop comprehensive library that makes explainable AI easy for data scientists, ML researchers and practitioners who need explanation for various types of data, models and explanation methods at different stages of ML process (data exploration, …
Interpreting Object-level Foundation Models via Visual Precision Search
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；School of Computing, NUS（新加坡国立大学计算机学院）；Huawei Technologies Co., Ltd.（华为技术有限公司）；College of Computer, NUDT（国防科技大学计算机学院）；Key Lab. of Edu. Inf. for Nationalities (YNNU), Ministry of Education（云南民族大学民族教育信息化教育部重点实验室）；School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区网络空间安全学院）
Unifying Image Counterfactuals and Feature Attributions with Latent-Space Adversarial Attacks
Department of Statistics, University of California, Berkeley（加州大学伯克利分校统计学系）；Department of Statistics and Data Science, University of Pennsylvania（宾夕法尼亚大学统计学与数据科学系）
UbiQTree: Uncertainty Quantification in XAI with Tree Ensembles
Center for Artificial Intelligence in Public Health Research (ZKI-PH), Robert Koch Institute（人工智能在公共卫生研究中的中心（ZKI-PH），罗伯特· Koch 研究所）；Department of Mathematics and Computer Science, Freie Universität Berlin（数学与计算机科学系，柏林自由大学）
University of Toronto（多伦多大学）；LG AI Research（LG AI研究院）
As an emerging field in Machine Learning, Explainable AI (XAI) has been offering remarkable performance in interpreting the decisions made by Convolutional Neural Networks (CNNs). To achieve visual explanations for CNNs, methods based on class activation mapping and randomized input sampling have gained great popularity. However, the attribution methods based on these techniques provide lower resolution and blurry explanation maps that limit their explanation power. To circumvent this issue, visualization based on various layers is sought. In this work, we collect visualization maps from multi…
Attri-Net: A Globally and Locally Inherently Interpretable Model for Multi-Label Classification Using Class-Specific Counterfactuals
Cluster of Excellence: Machine Learning - New Perspectives for Science, University of Tübingen（机器学习新视角科学卓越中心，图宾根大学）；Department of Computer Science, Friedrich-Alexander-Universität Erlangen-Nürnberg（计算机科学系，埃尔兰根-纽伦堡弗里德里希-亚历山大大学）；Hertie Institute for Artificial Intelligence in Brain Health, University of Tübingen（脑健康人工智能研究所，图宾根大学）；Department of Diabetes, Endocrinology, Nutritional Medicine and Metabolism, Inselspital, Bern University Hospital, University of Bern（糖尿病、内分泌学、营养医学和代谢学系，伯恩大学医院，伯恩大学）；Faculty of Health Sciences and Medicine, University of Lucerne（健康科学与医学学院，卢塞恩大学）
