Back propagation based visualizations have been proposed to interpret deep neural networks (DNNs), some of which produce interpretations with good visual quality. However, there exist doubts about whether these intuitive visualizations are related to the network decisions. Recent studies have confirmed this suspicion by verifying that almost all these modified back-propagation visualizations are not faithful to the model's decision-making process. Besides, these visualizations produce vague "relative importance scores", among which low values can't guarantee to be independent of the final prediction. Hence, it's highly desirable to develop a novel back-propagation framework that guarantees theoretical faithfulness and produces a quantitative attribution score with a clear understanding. To achieve the goal, we resort to mutual information theory to generate the interpretations, studying how much information of output is encoded in each input neuron. The basic idea is to learn a source signal by back-propagation such that the mutual information between input and output should be as much as possible preserved in the mutual information between input and the source signal. In addition, we propose a Mutual Information Preserving Inverse Network, termed MIP-IN, in which the parameters of each layer are recursively trained to learn how to invert. During the inversion, forward Relu operation is adopted to adapt the general interpretations to the specific input. We then empirically demonstrate that the inverted source signal satisfies completeness and minimality property, which are crucial for a faithful interpretation. Furthermore, the empirical study validates the effectiveness of interpretations generated by MIP-IN.
Towards Long-Horizon Interpretability: Efficient and Faithful Multi-Token Attribution for Reasoning LLMs
Department of Computer Science, City University of Hong Kong, Hong Kong SAR, China（香港城市大学计算机科学系）；Harbin Institute of Technology, Harbin, China（哈尔滨工业大学）
Attribution Explanations for Deep Neural Networks: A Theoretical Perspective
School of Computer Science and Technology, Xi’an Jiaotong University（西安交通大学计算机科学与技术学院）；School of Cyber Science and Engineering, Xi’an Jiaotong University（西安交通大学网络安全工程学院）；Department of Computer Science and Engineering, the John Hopcroft Center, Shanghai Jiao Tong University（上海交通大学计算机科学与工程系、霍夫曼中心）；Department of Data Science, New Jersey Institute of Technology（新泽西理工学院数据科学系）
本文研究了图神经网络（GNNs）的忠实解释。首先，我们提供了一个新的和一般的方法来正式表征GNN的解释的忠实性。它适用于现有的解释方法，包括特征属性和子图解释。其次，我们的分析和实证结果表明，特征属性的方法不能捕捉边缘特征的非线性效应，而现有的子图解释方法是不忠实的。第三，我们介绍了\emph{k-hop Explanation with a Convolutional Core}（KEC），一种新的解释方法，可证明最大限度地忠实于原来的GNN利用信息的图结构在其邻接矩阵和它的\emph{k-th}权力。最后，我们对使用GNN进行分类和异常检测任务的合成和真实数据集的实证结果证明了我们方法的有效性。
Ideal Attribution and Faithful Watermarks for Language Models
Data Science Institute, University of Chicago（芝加哥大学数据科学研究院）；Paul G. Allen School of Computer Science & Engineering, University of Washington（华盛顿大学保罗·G·阿伦计算机科学与工程学院）
Machine Learning Group, Technische Universität Berlin（柏林技术大学机器学习组）；BIFOLD – Berlin Institute for the Foundations of Learning and Data（柏林学习与数据基础研究院）
Activation patching is a standard method in mechanistic interpretability for localizing the components of a model responsible for specific behaviors, but it is computationally expensive to apply at scale. Attribution patching offers a faster, gradient-based approximation, yet suffers from noise and reduced reliability in deep, highly non-linear networks. In this work, we introduce Relevance Patching (RelP), which replaces the local gradients in attribution patching with propagation coefficients derived from Layer-wise Relevance Propagation (LRP). LRP propagates the network's output backward th…
Hidden Heroes and Gradient Bloats: Layer-Wise Redundancy Inverts Attribution in Transformers
Department of Computer Science, Fu Foundation School of Engineering and Applied Science, Columbia University, New York, NY, USA（计算机科学系，工程与应用科学基金会学院，哥伦比亚大学，纽约，纽约州，美国）
MFABA: A More Faithful and Accelerated Boundary-based Attribution Method for Deep Neural Networks
The University of Sydney（悉尼大学）；SuZhouYierqi（苏州义尔齐）；University of Malaya（马来亚大学）；CSIRO’s Data61（澳大利亚联邦科学与工业研究组织Data61）；Wayne State University（韦恩州立大学）；University of Texas at San Antonio（德克萨斯大学圣安东尼奥分校）
Various attribution methods have been developed to explain deep neural networks (DNNs) by inferring the attribution/importance/contribution score of each input variable to the final output. However, existing attribution methods are often built upon different heuristics. There remains a lack of a unified theoretical understanding of why these methods are effective and how they are related. To this end, for the first time, we formulate core mechanisms of fourteen attribution methods, which were designed on different heuristics, into the same mathematical system, i.e., the system of Taylor intera…
EAP-GP: Mitigating Saturation Effect in Gradient-based Automated Circuit Identification
King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）；University of Copenhagen（哥本哈根大学）；Peking University（北京大学）；University of Georgia（佐治亚大学）；Huazhong University of Science and Technology（华中科技大学）
Institute of Automation, CAS（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）
Modern deep learning models are notoriously opaque, which has motivated the development of methods for interpreting how deep models predict. This goal is usually approached with attribution method, which assesses the influence of features on model predictions. As an explanation method, the evaluation criteria of attribution methods is how accurately it re-reflects the actual reasoning process of the model (faithfulness). Meanwhile, since the reasoning process of deep models is inaccessible, researchers design various evaluation methods to demonstrate their arguments. However, some crucial logi…
Explaining Black-box Model Predictions via Two-level Nested Feature Attributions with Consistency Property
STAIR Lab, Chiba Institute of Technology（千叶技术学院STAIR实验室）；School of Mathematics and Statistics, The University of Melbourne（墨尔本大学数学与统计学学院）；ZOZO Research（ZOZO研究）
Less is More: Efficient Black-box Attribution via Minimal Interpretable Subset Selection
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；University of Chinese Academy of Sciences（中国科学院大学）；College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算机与数据科学学院）；School of Artificial Intelligence, University of Science and Technology Beijing（北京科技大学人工智能学院）；Department of Mechanical Engineering, Imperial College London（伦敦帝国理工学院机械工程系）；Center for Machine Vision and Signal Analysis (CMVS), University of Oulu（奥卢大学机器视觉与信号分析中心）；School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区计算机科学与技术学院）
AttnLRP: Attention-Aware Layer-Wise Relevance Propagation for Transformers
Fraunhofer Heinrich-Hertz-Institute（弗劳恩霍夫海因茨·赫兹研究所）；Technische Universität Berlin（柏林工业大学）；BIFOLD – Berlin Institute for the Foundations of Learning and Data（柏林学习与数据基础研究所（BIFOLD））
FRInGe: Distribution-Space Integrated Gradients with Fisher--Rao Geometry
Faculty of Computer Science / Doctoral School Computer Science University of Vienna（计算机科学学院/计算机科学博士学院维也纳大学）；ds:UniVie University of Vienna（ds:UniVie 维也纳大学）
