The study of neural networks from the perspective of Fourier features has garnered significant attention. While existing analytical research suggests that neural networks tend to learn low-frequency features, a clear attribution method for identifying the specific learned Fourier features has remained elusive. To bridge this gap, we propose a novel Fourier feature attribution method grounded in signal decomposition theory. Additionally, we analyze the differences between game-theoretic attribution metrics for Fourier and spatial domain features, demonstrating that game-theoretic evaluation metrics are better suited for Fourier-based feature attribution.
Our experiments show that Fourier feature attribution exhibits superior feature selection capabilities compared to spatial domain attribution methods. For instance, in the case of Vision Transformers (ViTs) on the ImageNet dataset, only $8\%$ of the Fourier features are required to maintain the original predictions for $80\%$ of the samples. Furthermore, we compare the specificity of features identified by our method against traditional spatial domain attribution methods. Results reveal that Fourier features exhibit greater intra-class concentration and inter-class distinctiveness, indicating their potential for more efficient classification and explainable AI algorithms.
Playing the network backward: A Game Theoretic Attribution Framework
Fraunhofer Heinrich-Hertz-Institute（弗劳恩霍夫海因里希-赫兹研究所）；Technische Universität Berlin（柏林技术大学）；Freie Universität Berlin（自由大学）；BIFOLD – Berlin Institute for the Foundations of Learning and Data（柏林学习与数据基础研究院）；Technological University Dublin（都柏林技术大学）
We compare machine learning explainability methods based on the theory of atomic (Shapley, 1953) and infinitesimal (Aumann and Shapley, 1974) games, in a theoretical and experimental investigation into how the model and choice of integration path can influence the resulting feature attributions. To gain insight into differences in attributions resulting from interventional Shapley values (Sundararajan and Najmi, 2019; Janzing et al., 2019; Chen et al., 2019) and Generalized Integrated Gradients (GIG) (Merrill et al., 2019) we note interventional Shapley is equivalent to a multi-path integratio…
One-class support vector machine (OC-SVM) for a long time has been one of the most effective anomaly detection methods and extensively adopted in both research as well as industrial applications. The biggest issue for OC-SVM is yet the capability to operate with large and high-dimensional datasets due to optimization complexity. Those problems might be mitigated via dimensionality reduction techniques such as manifold learning or autoencoder. However, previous work often treats representation learning and anomaly prediction separately. In this paper, we propose autoencoder based one-class supp…
In this paper, we study the problem of learning the exact structure of continuous-action games with non-parametric utility functions. We propose an $\ell_1$ regularized method which encourages sparsity of the coefficients of the Fourier transform of the recovered utilities. Our method works by accessing very few Nash equilibria and their noisy utilities. Under certain technical conditions, our method also recovers the exact structure of these utility functions, and thus, the exact structure of the game. Furthermore, our method only needs a logarithmic number of samples in terms of the number o…
QuadraSHAP: $ε$-Exact Shapley Values for Product Games in Logarithmic Parallel Time
AI Technology for Life, Information and Computing Sciences, Utrecht University, The Netherlands（荷兰乌得勒支大学人工智能技术与信息计算科学系）；Biomolecular Mass Spectrometry and Proteomics, Pharmaceutical Sciences, Utrecht University, The Netherlands（荷兰乌得勒支大学生物分子质谱与蛋白质组学，药学科学系）；Rational Intelligence Lab, CISPA Helmholtz Center for Information Security, Germany（德国CISPA海德堡信息安全中心理性智能实验室）；Epistemic Intelligence & Computation Lab, College of Computing & Data Science, Nanyang Technological University, Singapore（新加坡南洋理工大学认知智能与计算实验室）
A number of techniques have been proposed to explain a machine learning model's prediction by attributing it to the corresponding input features. Popular among these are techniques that apply the Shapley value method from cooperative game theory. While existing papers focus on the axiomatic motivation of Shapley values, and efficient techniques for computing them, they offer little justification for the game formulations used, and do not address the uncertainty implicit in their methods' outputs. For instance, the popular SHAP algorithm's formulation may give substantial attributions to featur…
Columbia University（哥伦比亚大学）；Stanford University（斯坦福大学）；Amazon AWS（亚马逊云科技）
Shapley value is a popular approach for measuring the influence of individual features. While Shapley feature attribution is built upon desiderata from game theory, some of its constraints may be less natural in certain machine learning settings, leading to unintuitive model interpretation. In particular, the Shapley value uses the same weight for all marginal contributions -- i.e. it gives the same importance when a large number of other features are given versus when a small number of other features are given. This property can be problematic if larger feature sets are more or less informati…
Who Does What in Deep Learning? Multidimensional Game-Theoretic Attribution of Function of Neural Units
Institute of Computational Neuroscience（计算神经科学研究所）；University Medical Center Eppendorf（埃本多夫大学医学中心）；International Max Planck Research School on Cognitive Neuroimaging（认知神经成像国际马克斯·普朗克研究学校）；MRC Cognition and Brain Sciences Unit（MRC认知与脑科学单位）；Mila - Quebec Artificial Intelligence Institute（魁北克人工智能研究所）；Learning in Machines & Brains（机器与大脑学习）；Departments of Bioengineering and Neuroscience（生物工程与神经科学系）；Department of Health Sciences（健康科学系）
A Hierarchy of Entropy-Shapley Games for Multivariate Predictive Uncertainty
Leibniz Institute for Prevention Research and Epidemiology – BIPS（莱布尼茨预防研究与流行病学研究所——BIPS）；University of Bremen（不来梅大学）；Simula Research Laboratory（西穆拉研究实验室）；Telenor Research & Innovation（挪威电信研究与创新部）；Norwegian Computing Center（挪威计算中心）
Feature attribution has been a foundational building block for explaining the input feature importance in supervised learning with Deep Neural Network (DNNs), but face new challenges when applied to deep Reinforcement Learning (RL).We propose a new approach to explaining deep RL actions by defining a class of \emph{action reconstruction} functions that mimic the behavior of a network in deep RL. This approach allows us to answer more complex explainability questions than direct application of DNN attribution methods, which we adapt to \emph{behavior-level attributions} in building our action r…
Technische Universität Berlin（柏林工业大学）；Zuse Institute Berlin（柏林祖斯研究所）
One of the goals of Explainable AI (XAI) is to determine which input components were relevant for a classifier decision. This is commonly know as saliency attribution. Characteristic functions (from cooperative game theory) are able to evaluate partial inputs and form the basis for theoretically "fair" attribution methods like Shapley values. Given only a standard classifier function, it is unclear how partial input should be realised. Instead, most XAI-methods for black-box classifiers like neural networks consider counterfactual inputs that generally lie off-manifold. This makes them hard to…
Additive feature explanations using Shapley values have become popular for providing transparency into the relative importance of each feature to an individual prediction of a machine learning model. While Shapley values provide a unique additive feature attribution in cooperative game theory, the Shapley values that can be generated for even a single machine learning model are far from unique, with theoretical and implementational decisions affecting the resulting attributions. Here, we consider the application of Shapley values for explaining decision tree ensembles and present a novel appro…
Fair feature attribution for multi-output prediction: a Shapley-based perspective
\, Chair of Computational Mathematics, DeustoTech, University of Deusto, Avenida de las Universidades 24, 48007 Bilbao, Basque Country, Spain \, Chair for dynamics, control, machine learning；numerics (Alexander Von Humboldt-Professorship), Department of Mathematics, Friedrich-Alexander-Universit\"at Erlangen-N\"urnberg, 91058, Erlangen, Germany. \, Universidad Aut\'onoma de Madrid, Departamento de Matem\'aticas, Ciudad Universitaria de Cantoblanco, 28049 Madrid, Spain. \, ATLAS Molecular Pharma, Parque Tecnol\'ogico de Bizkaia, Ed. 800, 48160 Derio, Spain. \, Precision Medicine；Metabolism Laboratory, CIC bioGUNE, Basque Research
School of Mathematics and Physics, The University of Queensland（昆士兰大学数学与物理学院）；SimulaMet, Simula Research Laboratory（SimulaMet、Simula研究院）；Department of Mathematics and Statistics, La Trobe University（拉筹伯大学数学与统计学院）
The Shapley value has become popular in the Explainable AI (XAI) literature, thanks, to a large extent, to a solid theoretical foundation, including four "favourable and fair" axioms for attribution in transferable utility games. The Shapley value is provably the only solution concept satisfying these axioms. In this paper, we introduce the Shapley value and draw attention to its recent uses as a feature selection tool. We call into question this use of the Shapley value, using simple, abstract "toy" counterexamples to illustrate that the axioms may work against the goals of feature selection.…
Interpreting predictions from tree ensemble methods such as gradient boosting machines and random forests is important, yet feature attribution for trees is often heuristic and not individualized for each prediction. Here we show that popular feature attribution methods are inconsistent, meaning they can lower a feature's assigned importance when the true impact of that feature actually increases. This is a fundamental problem that casts doubt on any comparison between features. To address it we turn to recent applications of game theory and develop fast exact tree solutions for SHAP (SHapley …
The Metagame of Interpretability and Meta-Attributions
University of Warsaw（华沙大学）；Centre for Credible AI, Warsaw University of Technology（可信AI中心，华沙技术大学）；Bielefeld University（比勒菲尔德大学）；LMU Munich, MCML（慕尼黑大学LMU，MCML）
When is the Computation of a Feature Attribution Method Tractable?
Institute for Mathematical and Computational Engineering, Universidad Católica de Chile（智利天主教大学数学与计算工程学院）；IMFD Chile（智利数学物理研究所）；CENIA Chile（智利国家人工智能研究中心）
University of California, Berkeley（加州大学伯克利分校）；Unveristy of California, San Diego（加州大学圣地亚哥分校）；Tsinghua University（清华大学）；Facebook AI Research（Facebook人工智能研究院）
Recent advances in multi-agent reinforcement learning (MARL) have achieved super-human performance in games like Quake 3 and Dota 2. Unfortunately, these techniques require orders-of-magnitude more training rounds than humans and don't generalize to new agent configurations even on the same game. In this work, we propose Collaborative Q-learning (CollaQ) that achieves state-of-the-art performance in the StarCraft multi-agent challenge and supports ad hoc team play. We first formulate multi-agent collaboration as a joint optimization on reward assignment and show that each agent has an approxim…
GISExplainer: On Explainability of Graph Neural Networks via Game-theoretic Interaction Subgraphs
School of Cyber Security and Information Law, Chongqing University of Posts and Telecommunications（重庆邮电大学网络空间安全与信息法学院）；School of Computer Science and Technology, Chongqing University of Posts and Telecommunications（重庆邮电大学计算机科学与技术学院）；Chengdu University of Technology（成都理工大学）；School of Software Engineering, Chengdu University of Information Technology（成都信息工程大学软件工程学院）；School of Computer and Information Science, Chongqing Normal University（重庆师范大学计算机与信息科学学院）
University of Washington（华盛顿大学）；Microsoft Research（微软研究院）
A variety of recent papers discuss the application of Shapley values, a concept for explaining coalitional games, for feature attribution in machine learning. However, the correct way to connect a machine learning model to a coalitional game has been a source of controversy. The two main approaches that have been proposed differ in the way that they condition on known features, using either (1) an interventional or (2) an observational conditional expectation. While previous work has argued that one of the two approaches is preferable in general, we argue that the choice is application depende…
ETH Zurich（苏黎世联邦理工学院）；Disney Research Zurich（苏黎世迪士尼研究中心）
The problem of explaining the behavior of deep neural networks has recently gained a lot of attention. While several attribution methods have been proposed, most come without strong theoretical foundations, which raises questions about their reliability. On the other hand, the literature on cooperative game theory suggests Shapley values as a unique way of assigning relevance scores such that certain desirable properties are satisfied. Unfortunately, the exact evaluation of Shapley values is prohibitively expensive, exponential in the number of input features. In this work, by leveraging recen…
Unveil Sources of Uncertainty: Feature Contribution to Conformal Prediction Intervals
Département de Mathématiques, Université du Québec à Montréal（魁北克大学蒙特利尔分校数学系）；Institut Intelligence et Données, Université Laval（拉瓦尔大学智能与数据研究所）；CNRS - Université de Montréal（法国国家科学研究中心-蒙特利尔大学）；CRM – CNRS Aix Marseille Univ, CNRS, AMSE, Marseille, France（法国国家科学研究中心-阿维尼昂大学）
Incentive salience attribution can be understood as a psychobiological mechanism ascribing relevance to potentially rewarding objects and actions. Despite being an important component of the motivational process guiding our everyday behaviour its study in naturalistic contexts is not straightforward. Here we propose a methodology based on artificial neural networks (ANNs) for approximating latent states produced by this process in situations where large volumes of behavioural data are available but no experimental control is possible. Leveraging knowledge derived from theoretical and computati…
Exact Shapley Attributions in Quadratic-time for FANOVA Gaussian Processes
Department of Computer Science, Vrije Universiteit Amsterdam, The Netherlands（荷兰阿姆斯特丹自由大学计算机科学系）；Rational Intelligence Lab, CISPA Helmholtz Center for Information Security, Germany（德国信息安全部CISPA理性智能实验室）；College of Computing & Data Science, Nanyang Technological University, Singapore（新加坡南洋理工大学计算与数据科学学院）
University of Oxford（牛津大学）；Microsoft Research Cambridge（微软剑桥研究院）
Feature attribution for kernel methods is often heuristic and not individualised for each prediction. To address this, we turn to the concept of Shapley values~(SV), a coalition game theoretical framework that has previously been applied to different machine learning model interpretation tasks, such as linear models, tree ensembles and deep networks. By analysing SVs from a functional perspective, we propose \textsc{RKHS-SHAP}, an attribution method for kernel machines that can efficiently compute both \emph{Interventional} and \emph{Observational Shapley values} using kernel mean embeddings o…
LMU Munich（慕尼黑大学）；Munich Center for Machine Learning（慕尼黑机器学习中心）；University of Warsaw（华沙大学）；Warsaw University of Technology（华沙理工大学）；Bielefeld University（比勒费尔德大学）；CITEC（认知交互技术中心）；Paderborn University（帕德博恩大学）
In imitation learning, it is common to learn a behavior policy to match an unknown target policy via max-likelihood training on a collected set of target demonstrations. In this work, we consider using offline experience datasets - potentially far from the target distribution - to learn low-dimensional state representations that provably accelerate the sample-efficiency of downstream imitation learning. A central challenge in this setting is that the unknown target policy itself may not exhibit low-dimensional behavior, and so there is a potential for the representation learning objective to a…
Generative Adversarial Networks (GANs) are powerful Machine Learning models capable of generating fully synthetic samples of a desired phenomenon with a high resolution. Despite their success, the training process of a GAN is highly unstable and typically it is necessary to implement several accessory heuristics to the networks to reach an acceptable convergence of the model. In this paper, we introduce a novel method to analyze the convergence and stability in the training of Generative Adversarial Networks. For this purpose, we propose to decompose the objective function of the adversary min…
We propose a principled method for kernel learning, which relies on a Fourier-analytic characterization of translation-invariant or rotation-invariant kernels. Our method produces a sequence of feature maps, iteratively refining the SVM margin. We provide rigorous guarantees for optimality and generalization, interpreting our algorithm as online equilibrium-finding dynamics in a certain two-player min-max game. Evaluations on synthetic and real-world datasets demonstrate scalability and consistent improvements over related random features-based methods.
Human emotion recognition has become a need for more realistic and interactive machines and computer systems. The greatest challenge is the availability of high-performance algorithms to effectively manage individual differences and nonstationarities in physiological data streams, i.e., algorithms that self-customize to a user with no subject-specific calibration data. We describe an evolving Gaussian Fuzzy Classifier (eGFC), which is supported by an online semi-supervised learning algorithm to recognize emotion patterns from electroencephalogram (EEG) data streams. We extract features from th…
The Distributional Uncertainty of the SHAP score in Explainable Machine Learning
Instituto de Ciencias de la Computación, UBA-CONICET（阿根廷计算机科学与技术研究所（布宜诺斯艾利斯大学-阿根廷国家科学与技术研究理事会联合机构））；Department of Computer Science, Universidad de Buenos Aires（布宜诺斯艾利斯大学计算机科学系）；Carleton Univ.（卡尔顿大学）；IMFD（智利数学物理研究所）；Universidad San Sebastián（圣塞瓦斯蒂安大学）；Department of Computer Science, University of Sheffield（谢菲尔德大学计算机科学系）；Artificial Intelligence Research Institute (IIIA-CSIC)（西班牙国家研究委员会人工智能研究所）；Universidad Católica de Chile（智利天主教大学）；CENIA Chile（智利国家环境技术中心）
