Attribution techniques explain the outcome of an AI model by assigning a numerical score to its inputs. So far, these techniques have mainly focused on attributing importance to static input features at a single point in time, and thus fail to generalize to sequential decision-making settings. This paper fills this gap by introducing techniques to generate attribution-based explanations for Markov Decision Processes (MDPs). We give a formal characterization of what attributions should represent in MDPs, focusing on explanations that assign importance scores to both individual states and execution paths. We show how importance scores can be computed by leveraging techniques for strategy synthesis, enabling the efficient computation of these scores despite the non-determinism inherent in an MDP. We evaluate our approach on five case-studies, demonstrating its utility in providing interpretable insights into the logic of sequential decision-making agents.
Structural Compactness as a Complementary Criterion for Explanation Quality
Fraunhofer Heinrich Hertz Institute(弗劳恩霍夫海因里希·赫兹研究所);Technische Universität Berlin(柏林工业大学);BIFOLD – Berlin Institute for the Foundations of Learning and Data(BIFOLD – 柏林学习与数据基础研究所);Centre of eXplainable Artificial Intelligence, Technological University Dublin(都柏林科技大学可解释人工智能中心)
An Empirical Evaluation of Factors Affecting SHAP Explanation of Time Series Classification
School of Engineering Mathematics and Technology, University of Bristol, UK(工程数学与技术学院,布里斯托大学);School of Computer Science, University College Dublin, Ireland(计算机科学学院,都柏林大学学院)
Provably Better Explanations with Optimized Aggregation of Feature Attributions
LMU Munich(慕尼黑大学);Siemens AG(西门子公司);Technical University of Munich(慕尼黑工业大学);University of Oxford(牛津大学);Munich Center for Machine Learning (MCML)(慕尼黑机器学习中心);German Cancer Research Center (DKFZ)(德国癌症研究中心);Goethe University Frankfurt(法兰克福大学)
From Graphs to Gradients: Physics-Inspired Structural Attribution for Cyber-Physical IoT Systems and Beyond
Research & Innovation Development Department, Netcompany S.A.(研究与创新发展部,Netcompany S.A.公司);Department of Informatics and Telematics, Harokopio University of Athens(雅典哈罗科皮奥大学信息学与电信系);Department of Electrical and Computer Engineering, Hellenic Mediterranean University(希腊地中海大学电气与计算机工程系);Department of Electrical and Computer Engineering, University of Western Macedonia(西马其顿大学电气与计算机工程系)
"Is your explanation stable?": A Robustness Evaluation Framework for Feature Attribution
Zhejiang University(浙江大学);ETH Zürich(苏黎世联邦理工学院);The Pennsylvania State University(宾夕法尼亚州立大学)
理解神经网络的决策过程是困难的。解释的一个重要方法是将其决定性归因于关键特征。虽然有许多算法被提出,但大多数算法只是提高了对模型的忠实度。然而,现实环境中存在大量的随机噪声,这可能导致解释的波动。更严重的是,最近的工作表明,解释算法容易受到对抗攻击。所有这些都使得解释在真实场景中难以信任。为了弥补这一差距,我们提出了一个模型无关的方法\emph{Median Test for Feature Attribution}(MeTFA)量化的不确定性,并增加了理论保证的解释算法的稳定性。MeTFA具有以下两个功能:(1)检查一个特征是否显著重要或不重要,并生成MeTFA显著图以可视化结果;(2)计算特征归因得分的置信区间,并生成MeTFA平滑图以增加解释的稳定性。实验结果表明,MeTFA提高了解释的视觉质量,在保持忠实性的同时,显著降低了解释的不稳定性。为了定量评估不同噪声环境下的解释的忠实性,我们进一步提出了几个鲁棒的忠实性度量。实验结果表明,MeTFA平滑解释可以显着提高鲁棒忠实性。此外,我们使用两个场景来展示MeTFA在应用中的潜力。首先,当应用于SOTA解释方法来定位语义分割模型的上下文偏差时,MeTFA显著解释使用小得多的区域来保持99%以上的忠实度。第二,当用不同的解释导向攻击进行测试时,MeTFA可以帮助防御普通攻击,以及自适应的对抗攻击。
Attribution Explanations for Deep Neural Networks: A Theoretical Perspective
School of Computer Science and Technology, Xi’an Jiaotong University(西安交通大学计算机科学与技术学院);School of Cyber Science and Engineering, Xi’an Jiaotong University(西安交通大学网络安全工程学院);Department of Computer Science and Engineering, the John Hopcroft Center, Shanghai Jiao Tong University(上海交通大学计算机科学与工程系、霍夫曼中心);Department of Data Science, New Jersey Institute of Technology(新泽西理工学院数据科学系)
Explanation Bias is a Product: Revealing the Hidden Lexical and Position Preferences in Post-Hoc Feature Attribution
Computational Linguistics and Text Mining Lab, Vrije Universiteit Amsterdam(计算语言学与文本挖掘实验室,荷兰自由大学);TNO–The Netherlands Organization for Applied Scientific Research, The Hague(荷兰应用科学研究院(TNO),海牙);Leiden University Centre for Linguistics (LUCL), Leiden University, Leiden(莱顿大学语言中心(LUCL),莱顿大学,莱顿)
Towards A Unified Information Bottleneck Framework for Time Series Explanations
Florida International University(佛罗里达国际大学);Carnegie Mellon University(卡内基梅隆大学);University of Houston(休斯顿大学);NEC Labs America(美国 NEC 实验室);Singapore Management University(新加坡管理大学)
Explanations of Large Language Models Explain Language Representations in the Brain
Iran University of Science and Technology(伊朗科学技术大学);University of Tehran(德黑兰大学);Institute for Research in Fundamental Sciences(基础科学研究所);Khatam University(哈塔姆大学)
From Attribution Maps to Human-Understandable Explanations through Concept Relevance Propagation
Fraunhofer Heinrich-Hertz-Institute(弗劳恩霍夫海因里希-赫兹研究所);Technische Universität Berlin(柏林工业大学);BIFOLD – Berlin Institute for the Foundations of Learning and Data(BIFOLD – 柏林学习与数据基础研究所)
本文研究了图神经网络(GNNs)的忠实解释。首先,我们提供了一个新的和一般的方法来正式表征GNN的解释的忠实性。它适用于现有的解释方法,包括特征属性和子图解释。其次,我们的分析和实证结果表明,特征属性的方法不能捕捉边缘特征的非线性效应,而现有的子图解释方法是不忠实的。第三,我们介绍了\emph{k-hop Explanation with a Convolutional Core}(KEC),一种新的解释方法,可证明最大限度地忠实于原来的GNN利用信息的图结构在其邻接矩阵和它的\emph{k-th}权力。最后,我们对使用GNN进行分类和异常检测任务的合成和真实数据集的实证结果证明了我们方法的有效性。
HiLRP: Toward One Trustworthy Explanation for Vision Transformer: Conservation-Valid Attribution via Attention Primitives
Faculty of Engineering, University of Peradeniya(佩拉德尼亚大学工程学院);School of Computing Technologies, RMIT University(皇家墨尔本理工大学计算技术学院);School of Engineering & Digital Technologies, University of Southern Queensland(南昆士兰大学工程与数字技术学院);School of Engineering & Technologies, UNSW(新南威尔士大学工程与技术学院);Faculty of Science and Technology, Charles Darwin University(查尔斯达尔文大学科学技术学院)
Revealing Hidden Context Bias in Segmentation and Object Detection through Concept-specific Explanations
Fraunhofer Heinrich-Hertz-Institute(弗劳恩霍夫海因里希·赫兹研究所);Technische Universität Berlin(柏林工业大学);Berlin Institute for the Foundations of Learning and Data(柏林学习与数据基础研究所)