In this paper, we consider a perturbation-based metric of predictive faithfulness of feature rankings (or attributions) that we call PGI squared. When applied to decision tree-based regression models, the metric can be computed accurately and efficiently for arbitrary independent feature perturbation distributions. In particular, the computation does not involve Monte Carlo sampling that has been typically used for computing similar metrics and which is inherently prone to inaccuracies. Moreover, we propose a method of ranking features by their importance for the tree model's predictions based on PGI squared. Our experiments indicate that in some respects, the method may identify the globally important features better than the state-of-the-art SHAP explainer
Mila – Quebec AI Institute（米拉-魁北克人工智能研究所）；Polytechnique Montréal（蒙特利尔综合理工学院）；McGill University（麦吉尔大学）；Facebook（脸书）
To explain NLP models a popular approach is to use importance measures, such as attention, which inform input tokens are important for making a prediction. However, an open question is how well these explanations accurately reflect a model's logic, a property called faithfulness. To answer this question, we propose Recursive ROAR, a new faithfulness metric. This works by recursively masking allegedly important tokens and then retraining the model. The principle is that this should result in worse model performance compared to masking random tokens. The result is a performance curve given a mas…
Faithfulness Without Anchors: A Position Paper on the Structural Misalignments of Explainable AI Faithfulness Metrics for Feature Importance
TU Delft（代尔夫特理工大学）；XITASO GmbH（XITASO有限公司）；University College Cork（科克大学学院）；RPTU University Kaiserslautern-Landau（凯撒斯劳滕-兰道工业大学）；DFKI（德国人工智能研究中心）；TU Dresden（德累斯顿理工大学）
Towards Long-Horizon Interpretability: Efficient and Faithful Multi-Token Attribution for Reasoning LLMs
Department of Computer Science, City University of Hong Kong, Hong Kong SAR, China（香港城市大学计算机科学系）；Harbin Institute of Technology, Harbin, China（哈尔滨工业大学）
Cooperative Institute for Severe and High-Impact Weather Research and Operations, University of Oklahoma（俄克拉荷马大学强天气与高影响天气合作研究所）；University of Oklahoma（俄克拉荷马大学）；NOAA/OAR/National Severe Storms Laboratory（美国国家海洋和大气管理局/海洋与大气研究办公室/国家强风暴实验室）；Verisk Analytics Incorporated（Verisk分析公司）；NSF AI Institute for Research on Trustworthy AI in Weather, Climate, and Coastal Oceanography（美国国家科学基金会天气、气候与沿海海洋学可信人工智能研究人工智能研究所）
Machine learning (ML) models are becoming increasingly common in the atmospheric science community with a wide range of applications. To enable users to understand what an ML model has learned, ML explainability has become a field of active research. In Part I of this two-part study, we described several explainability methods and demonstrated that feature rankings from different methods can substantially disagree with each other. It is unclear, though, whether the disagreement is overinflated due to some methods being less faithful in assigning importance. Herein, "faithfulness" or "fidelity"…
City University of Hong Kong（香港城市大学）；National University of Singapore（新加坡国立大学）；The Hong Kong Polytechnic University（香港理工大学）
Attention mechanisms are dominating the explainability of deep models. They produce probability distributions over the input, which are widely deemed as feature-importance indicators. However, in this paper, we find one critical limitation in attention explanations: weakness in identifying the polarity of feature impact. This would be somehow misleading -- features with higher attention weights may not faithfully contribute to model predictions; instead, they can impose suppression effects. With this finding, we reflect on the explainability of current attention-based techniques, such as Atten…
University of Warsaw（华沙大学）；IDEAS NCBR；MIM Solutions
Shapley values are one of the main tools used to explain predictions of tree ensemble models. The main alternative to Shapley values are Banzhaf values that have not been understood equally well. In this paper we make a step towards filling this gap, providing both experimental and theoretical comparison of these model explanation methods. Surprisingly, we show that Banzhaf values offer several advantages over Shapley values while providing essentially the same explanations. We verify that Banzhaf values: (1) have a more intuitive interpretation, (2) allow for more efficient algorithms, and (3…
Department of Statistical Science, Cornell University（康奈尔大学统计科学系）
We propose a modification that corrects for split-improvement variable importance measures in Random Forests and other tree-based methods. These methods have been shown to be biased towards increasing the importance of features with more potential splits. We show that by appropriately incorporating split-improvement as measured on out of sample data, this bias can be corrected yielding better summaries and screening tools.
We attempt to give a unifying view of the various recent attempts to (i) improve the interpretability of tree-based models and (ii) debias the the default variable-importance measure in random Forests, Gini importance. In particular, we demonstrate a common thread among the out-of-bag based bias correction methods and their connection to local explanation for trees. In addition, we point out a bias caused by the inclusion of inbag data in the newly developed explainable AI for trees algorithms.
The recent increase in the deployment of machine learning models in critical domains such as healthcare, criminal justice, and finance has highlighted the need for trustworthy methods that can explain these models to stakeholders. Feature importance methods (e.g. gain and SHAP) are among the most popular explainability methods used to address this need. For any explainability technique to be trustworthy and meaningful, it has to provide an explanation that is accurate and stable. Although the stability of local feature importance methods (explaining individual predictions) has been studied bef…
This paper introduces Tree-Pyramidal Adaptive Importance Sampling (TP-AIS), a novel iterated sampling method that outperforms state-of-the-art approaches like deterministic mixture population Monte Carlo (DM-PMC), mixture population Monte Carlo (M-PMC) and layered adaptive importance sampling (LAIS). TP-AIS iteratively builds a proposal distribution parameterized by a tree pyramid, where each tree leaf spans a subspace that represents its importance density. After each new sample operation, a set of tree leaves are subdivided for improving the approximation of the proposal distribution to the …
Current limitations in boosted tree modelling prevent the effective scaling to datasets with a large feature number, particularly when investigating the magnitude and directionality of various features on classification. We present a novel methodology, Hollow-tree Super (HOTS), to resolve and visualize feature importance in boosted tree models involving a large number of features. Further, HOTS allows for investigation of the directionality and magnitude various features have on classification. Using the Iris dataset, we first compare HOTS to Gini Importance, Partial Dependence Plots, and Perm…
Tree ensembles such as Random Forests have achieved impressive empirical success across a wide variety of applications. To understand how these models make predictions, people routinely turn to feature importance measures calculated from tree ensembles. It has long been known that Mean Decrease Impurity (MDI), one of the most widely used measures of feature importance, incorrectly assigns high importance to noisy features, leading to systematic bias in feature selection. In this paper, we address the feature selection bias of MDI from both theoretical and methodological perspectives. Based on …
Gradient Boosting Machines (GBM) are among the go-to algorithms on tabular data, which produce state of the art results in many prediction tasks. Despite its popularity, the GBM framework suffers from a fundamental flaw in its base learners. Specifically, most implementations utilize decision trees that are typically biased towards categorical variables with large cardinalities. The effect of this bias was extensively studied over the years, mostly in terms of predictive performance. In this work, we extend the scope and study the effect of biased base learners on GBM feature importance (FI) m…
We study the problem of interpreting trained classification models in the setting of linguistic data sets. Leveraging a parse tree, we propose to assign least-squares based importance scores to each word of an instance by exploiting syntactic constituency structure. We establish an axiomatic characterization of these importance scores by relating them to the Banzhaf value in coalitional game theory. Based on these importance scores, we develop a principled method for detecting and quantifying interactions between words in a sentence. We demonstrate that the proposed method can aid in interpret…
Local MDI+: Local Feature Importances for Tree-Based Models
Department of Computational Precision Health（计算精准健康系）；UC Berkeley（伯克利大学）；UCSF（旧金山分校）；Department of Statistics（统计系）；Department of Applied and Computational Mathematics and Statistics（应用与计算数学和统计系）；University of Notre Dame（诺特大学）；Department of Statistics and EECS（统计系和电子工程与计算机科学系）
Université d’Abomey-Calavi（阿波美卡拉维大学）；Université Paris 1 Panthéon Sorbonne（巴黎第一大学（先贤祠-索邦大学））
This paper deals with prediction of anopheles number, the main vector of malaria risk, using environmental and climate variables. The variables selection is based on an automatic machine learning method using regression trees, and random forests combined with stratified two levels cross validation. The minimum threshold of variables importance is accessed using the quadratic distance of variables importance while the optimal subset of selected variables is used to perform predictions. Finally the results revealed to be qualitatively better, at the selection, the prediction , and the CPU time p…
Duke University（杜克大学）；Nvidia Corporation（英伟达公司）；National Taiwan University of Science and Technology（台湾科技大学）；Texas A&M University（得克萨斯农工大学）
Design flow parameters are of utmost importance to chip design quality and require a painfully long time to evaluate their effects. In reality, flow parameter tuning is usually performed manually based on designers' experience in an ad hoc manner. In this work, we introduce a machine learning-based automatic parameter tuning methodology that aims to find the best design quality with a limited number of trials. Instead of merely plugging in machine learning engines, we develop clustering and approximate sampling techniques for improving tuning efficiency. The feature extraction in this method c…
Twitter（推特）；Google（谷歌）；Airbnb（爱彼迎）；Facebook（脸书）；Penn State University（宾夕法尼亚州立大学）
Historical features are important in ads click-through rate (CTR) prediction, because they account for past engagements between users and ads. In this paper, we study how to efficiently construct historical features through counting features. The key challenge of such problem lies in how to automatically identify counting keys. We propose a tree-based method for counting key selection. The intuition is that a decision tree naturally provides various combinations of features, which could be used as counting key candidate. In order to select personalized counting features, we train one decision …
SINTEF DIGITAL（挪威科学与工业研究基金会数字研究所）；Norwegian University of Science and Technology（挪威科技大学）；SimulaMet（西穆拉代谢研究所）；Yale School of Public Health（耶鲁大学公共卫生学院）
Estimating feature importance is a significant aspect of explaining data-based models. Besides explaining the model itself, an equally relevant question is which features are important in the underlying data generating process. We present a Shapley value based framework for inferring the importance of individual features, including uncertainty in the estimator. We build upon the recently published feature importance measure of SAGE (Shapley additive global importance) and introduce sub-SAGE which can be estimated without resampling for tree-based models. We argue that the uncertainties can be …
Paul G. Allen School of Computer Science, University of Washington（华盛顿大学保罗·G·艾伦计算机科学学院）；Department of Genome Sciences, University of Washington（华盛顿大学基因组科学系）
Note that a newer expanded version of this paper is now available at: arXiv:1802.03888 It is critical in many applications to understand what features are important for a model, and why individual predictions were made. For tree ensemble methods these questions are usually answered by attributing importance values to input features, either globally or for a single prediction. Here we show that current feature attribution methods are inconsistent, which means changing the model to rely more on a given feature can actually decrease the importance assigned to that feature. To address this problem…
University of Edinburgh（爱丁堡大学）；University of Amsterdam（阿姆斯特丹大学）
We treat projective dependency trees as latent variables in our probabilistic model and induce them in such a way as to be beneficial for a downstream task, without relying on any direct tree supervision. Our approach relies on Gumbel perturbations and differentiable dynamic programming. Unlike previous approaches to latent tree learning, we stochastically sample global structures and our parser is fully differentiable. We illustrate its effectiveness on sentiment analysis and natural language inference tasks. We also study its properties on a synthetic structure induction task. Ablation studi…
FitCF: A Framework for Automatic Feature Importance-guided Counterfactual Example Generation
Quality and Usability Lab, Technische Universität Berlin（柏林技术大学质量与可用性实验室）；German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）；Deutsche Telekom（德国电信）；Saarland Informatics Campus（萨尔州信息学校区）；Centre for European Research in Trusted AI (CERTAIN)（可信人工智能欧洲研究中心）；BIFOLD – Berlin Institute for the Foundations of Learning and Data（柏林学习与数据基础研究院）
University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）
Many past works aim to improve visual reasoning in models by supervising feature importance (estimated by model explanation techniques) with human annotations such as highlights of important image regions. However, recent work has shown that performance gains from feature importance (FI) supervision for Visual Question Answering (VQA) tasks persist even with random supervision, suggesting that these methods do not meaningfully align model FI with human FI. In this paper, we show that model FI supervision can meaningfully improve VQA model accuracy as well as performance on several Right-for-th…
King’s College London（伦敦国王学院）；Turing Intelligence Technology Limited（图灵智能科技有限公司）
Feature importance aims at measuring how crucial each input feature is for model prediction. It is widely used in feature engineering, model selection and explainable artificial intelligence (XAI). In this paper, we propose a new tree-model explanation approach for model selection. Our novel concept leverages the Coefficient of Variation of a feature weight (measured in terms of the contribution of the feature to the prediction) to capture the dispersion of importance over samples. Extensive experimental results show that our novel feature explanation performs better than general cross validat…
Jožef Stefan Institute（约瑟夫·斯特凡研究所）；Jožef Stefan International Postgraduate School（约瑟夫·斯特凡国际研究生院）
Predictive clustering trees (PCTs) are a well established generalization of standard decision trees, which can be used to solve a variety of predictive modeling tasks, including structured output prediction. Combining them into ensembles yields state-of-the-art performance. Furthermore, the ensembles of PCTs can be interpreted by calculating feature importance scores from the learned models. However, their learning time scales poorly with the dimensionality of the output space. This is often problematic, especially in (hierarchical) multi-label classification, where the output can consist of h…
Local explanations of learning-to-rank (LTR) models are thought to extract the most important features that contribute to the ranking predicted by the LTR model for a single data point. Evaluating the accuracy of such explanations is challenging since the ground truth feature importance scores are not available for most modern LTR models. In this work, we propose a systematic evaluation technique for explanations of LTR models. Instead of using black-box models, such as neural networks, we propose to focus on tree-based LTR models, from which we can extract the ground truth feature importance …
Tree ensembles, such as random forests and AdaBoost, are ubiquitous machine learning models known for achieving strong predictive performance across a wide variety of domains. However, this strong performance comes at the cost of interpretability (i.e. users are unable to understand the relationships a trained random forest has learned and why it is making its predictions). In particular, it is challenging to understand how the contribution of a particular feature, or group of features, varies as their value changes. To address this, we introduce Disentangled Attribution Curves (DAC), a method…
Augmented Functional Random Forests: Classifier Construction and Unbiased Functional Principal Components Importance through Ad-Hoc Conditional Permutations
Faculty of Technological and Innovation Sciences（技术与创新科学学院）；Faculty of Economics and Law（经济与法学院）
Tree ensembles distribute feature importance evenly amongst groups of correlated features. The average feature ranking of the correlated group is suppressed, which reduces interpretability and complicates feature selection. In this paper we present ControlBurn, a feature selection algorithm that uses a weighted LASSO-based feature selection method to prune unnecessary features from tree ensembles, just as low-intensity fire reduces overgrown vegetation. Like the linear LASSO, ControlBurn assigns all the feature importance of a correlated group of features to a single feature. Moreover, the alg…
Hamilton Institute（汉密尔顿研究所）；Maynooth University（梅努斯大学）
We develop a new approach for feature selection via gain penalization in tree-based models. First, we show that previous methods do not perform sufficient regularization and often exhibit sub-optimal out-of-sample performance, especially when correlated features are present. Instead, we develop a new gain penalization idea that exhibits a general local-global regularization for tree-based models. The new method allows for more flexibility in the choice of feature-specific importance weights. We validate our method on both simulated and real data and implement itas an extension of the popular R…
Graph Neural Networks (GNNs) have achieved significant success in learning better representations by performing feature propagation and transformation iteratively to leverage neighborhood information. Nevertheless, iterative propagation restricts the information of higher-layer neighborhoods to be transported through and fused with the lower-layer neighborhoods', which unavoidably results in feature smoothing between neighborhoods in different layers and can thus compromise the performance, especially on heterophily networks. Furthermore, most deep GNNs only recognize the importance of higher-…
Dutch Research Organization for Applied Research (TNO)（荷兰应用科学研究组织（TNO））；Technical University of Delft（代尔夫特理工大学）；Utrecht University（乌得勒支大学）
Recent advances in interpretable Machine Learning (iML) and eXplainable AI (XAI) construct explanations based on the importance of features in classification tasks. However, in a high-dimensional feature space this approach may become unfeasible without restraining the set of important features. We propose to utilize the human tendency to ask questions like "Why this output (the fact) instead of that output (the foil)?" to reduce the number of features to those that play a main role in the asked contrast. Our proposed method utilizes locally trained one-versus-all decision trees to identify th…
