Explainable artificial intelligence (XAI) methods are currently evaluated with approaches mostly originated in interpretable machine learning (IML) research that focus on understanding models such as comparison against existing attribution approaches, sensitivity analyses, gold set of features, axioms, or through demonstration of images. There are problems with these methods such as that they do not indicate where current XAI approaches fail to guide investigations towards consistent progress of the field. They do not measure accuracy in support of accountable decisions, and it is practically impossible to determine whether one XAI method is better than the other or what the weaknesses of existing models are, leaving researchers without guidance on which research questions will advance the field. Other fields usually utilize ground-truth data and create benchmarks. Data representing ground-truth explanations is not typically used in XAI or IML. One reason is that explanations are subjective, in the sense that an explanation that satisfies one user may not satisfy another. To overcome these problems, we propose to represent explanations with canonical equations that can be used to evaluate the accuracy of XAI methods. The contributions of this paper include a methodology to create synthetic data representing ground-truth explanations, three data sets, an evaluation of LIME using these data sets, and a preliminary analysis of the challenges and potential benefits in using these data to evaluate existing XAI approaches. Evaluation methods based on human-centric studies are outside the scope of this paper.
In this work, we report the practical and theoretical aspects of Explainable AI (XAI) identified in some fundamental literature. Although there is a vast body of work on representing the XAI backgrounds, most of the corpuses pinpoint a discrete direction of thoughts. Providing insights into literature in practice and theory concurrently is still a gap in this field. This is important as such connection facilitates a learning process for the early stage XAI researchers and give a bright stand for the experienced XAI scholars. Respectively, we first focus on the categories of black-box explanati…
Scrutinizing XAI using linear ground-truth data with suppressor variables
Technische Universität Berlin（柏林工业大学）；Eindhoven University of Technology（埃因霍温理工大学）；BIFOLD – Berlin Institute for the Foundations of Learning and Data（柏林学习与数据基础研究所（BIFOLD））；Korea University（高丽大学）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；Physikalisch-Technische Bundesanstalt（德国联邦物理技术研究院）；Charité – Universitätsmedizin Berlin（柏林夏里特医学院）
Mind the XAI Gap: A Human-Centered LLM Framework for Democratizing Explainable AI
Aristotle University of Thessaloniki, School of Informatics（塞萨洛尼基亚里士多德大学信息学院）；Technical University of Denmark（丹麦技术大学）；Telefonica Scientific Research（电信科学研究院）
Explainable artificial intelligence (XAI) methods lack ground truth. In its place, method developers have relied on axioms to determine desirable properties for their explanations' behavior. For high stakes uses of machine learning that require explainability, it is not sufficient to rely on axioms as the implementation, or its usage, can fail to live up to the ideal. As a result, there exists active research on validating the performance of XAI methods. The need for validation is especially magnified in domains with a reliance on XAI. A procedure frequently used to assess their utility, and t…
EXplainable AI (XAI) methods have been proposed to interpret how a deep neural network predicts inputs through model saliency explanations that highlight the parts of the inputs deemed important to arrive a decision at a specific target. However, it remains challenging to quantify correctness of their interpretability as current evaluation approaches either require subjective input from humans or incur high computation cost with automated evaluation. In this paper, we propose backdoor trigger patterns--hidden malicious functionalities that cause misclassification--to automate the evaluation of…
A Differentiable Atari VCS:A Complex, Fully Known Ground Truth for Explainable AI
Pattern Recognition Lab, Friedrich-Alexander-University Erlangen-Nuremberg（模式识别实验室，弗里德里希-亚历山大-埃森纳赫大学）；Mannheim Center for Neuromodulation and Neuroprosthetics, Heidelberg University（曼海姆神经调制与神经假体中心，海德堡大学）
Finding the right XAI method -- A Guide for the Evaluation and Ranking of Explainable AI Methods in Climate Science
Technische Universität Berlin（柏林工业大学）；ATB（莱布尼茨农业工程与生物经济研究所）；University of Leipzig（莱比锡大学）；University of Reading（雷丁大学）；BIFOLD – Berlin Institute for the Foundations of Learning and Data（柏林学习与数据基础研究所（BIFOLD））；UiT the Arctic University of Norway（挪威北极大学）；University of Potsdam（波茨坦大学）
From Flexibility to Manipulation: The Slippery Slope of XAI Evaluation
UiT The Arctic University of Norway（挪威北极大学）；Leibniz Institute of Agricultural Engineering and Bioeconomy e.V. (ATB)（莱布尼茨农业工程与生物经济研究所）；University of Potsdam（波茨坦大学）；BIFOLD - Berlin Institute for the Foundations of Learning and Data（柏林学习与数据基础研究所）；TU Berlin（柏林工业大学）；Fraunhofer HHI（弗劳恩霍夫海因里希·赫兹研究所）
Simon Fraser University（西蒙弗雷泽大学）；The University of British Columbia（不列颠哥伦比亚大学）
Explainable artificial intelligence (XAI) is essential for enabling clinical users to get informed decision support from AI and comply with evidence-based medical practice. Applying XAI in clinical settings requires proper evaluation criteria to ensure the explanation technique is both technically sound and clinically useful, but specific support is lacking to achieve this goal. To bridge the research gap, we propose the Clinical XAI Guidelines that consist of five criteria a clinical XAI needs to be optimized for. The guidelines recommend choosing an explanation form based on Guideline 1 (G1)…
The Meta-Evaluation Problem in Explainable AI: Identifying Reliable Estimators with MetaQuantus
Technical University of Berlin（柏林工业大学）；Berlin Institute for the Foundations of Learning and Data（柏林学习与数据基础研究所）；UiT Arctic University of Norway（挪威北极大学）；Fraunhofer Heinrich-Hertz-Institute（弗劳恩霍夫海因里希·赫兹研究所）；University of Potsdam（波茨坦大学）；Leibniz Institute of Agricultural Engineering and Bioeconomy e.V.（莱布尼茨农业工程与生物经济研究所）
University College Dublin（都柏林大学）；Insight Centre for Data Analytics（洞察数据分析中心）；VistaMilk SFI Research Centre（VistaMilk科学基金会爱尔兰研究中心）
Counterfactual explanations have emerged as a popular solution for the eXplainable AI (XAI) problem of elucidating the predictions of black-box deep-learning systems due to their psychological validity, flexibility across problem domains and proposed legal compliance. While over 100 counterfactual methods exist, claiming to generate plausible explanations akin to those preferred by people, few have actually been tested on users ($\sim7\%$). So, the psychological validity of these counterfactual algorithms for effective XAI for image data is not established. This issue is addressed here using a…
Understanding the reasons behind the predictions made by deep neural networks is critical for gaining human trust in many important applications, which is reflected in the increasing demand for explainability in AI (XAI) in recent years. Saliency-based feature attribution methods, which highlight important parts of images that contribute to decisions by classifiers, are often used as XAI methods, especially in the field of computer vision. In order to compare various saliency-based XAI methods quantitatively, several approaches for automated evaluation schemes have been proposed; however, ther…
Why Do Class-Dependent Evaluation Effects Occur with Time Series Feature Attributions? A Synthetic Data Investigation
Information Systems, Eindhoven University of Technology, Eindhoven, The Netherlands（信息系统，埃因霍温理工大学，埃因霍温，荷兰）；Eindhoven Artificial Intelligence Systems Institute, Eindhoven University of Technology, Eindhoven, The Netherlands（埃因霍温人工智能系统研究所，埃因霍温理工大学，埃因霍温，荷兰）；Human-Technology Interaction, Eindhoven University of Technology, Eindhoven, The Netherlands（人机交互，埃因霍温理工大学，埃因霍温，荷兰）
Does Explanation Correctness Matter? Linking Computational XAI Evaluation to Human Understanding
Information Systems Group, Eindhoven University of Technology（埃因霍温理工大学信息系统组）；Human-Technology Interaction Group, Eindhoven University of Technology（埃因霍温理工大学人机交互组）
Classification Metrics for Image Explanations: Towards Building Reliable XAI-Evaluations
Fraunhofer Institute for Manufacturing Engineering and Automation IPA（弗劳恩霍夫制造工程与自动化研究所）；Institute of Industrial Manufacturing and Management IFF, University of Stuttgart（斯图加特大学工业制造与管理研究所）
Explainable AI has attracted much research attention in recent years with feature attribution algorithms, which compute "feature importance" in predictions, becoming increasingly popular. However, there is little analysis of the validity of these algorithms as there is no "ground truth" in the existing datasets to validate their correctness. In this work, we develop a method to quantitatively evaluate the correctness of XAI algorithms by creating datasets with known explanation ground truth. To this end, we focus on the binary classification problems. String datasets are constructed using form…
Explaining AI Decisions: Towards Achieving Human-Centered Explainability in Smart Home Environments
Fraunhofer Institute for Applied Information Technology FIT（弗劳恩霍夫应用信息技术研究所）；University of Siegen（锡根大学）；Bonn-Rhein-Sieg University of Applied Sciences（波恩-莱茵-锡格应用科学大学）
The game industry has long been troubled by malicious activities utilizing game bots. The game bots disturb other game players and destroy the environmental system of the games. For these reasons, the game industry put their best efforts to detect the game bots among players' characters using the learning-based detections. However, one problem with the detection methodologies is that they do not provide rational explanations about their decisions. To resolve this problem, in this work, we investigate the explainabilities of the game bot detection. We develop the XAI model using a dataset from …
Robustness of Explainable Artificial Intelligence in Industrial Process Modelling
Technical University Graz（格拉茨工业大学）；voestalpine Stahl GmbH（奥钢联钢铁股份公司）；K1-MET GmbH（K1-MET有限公司）；Christian Doppler Laboratory for Dependable Intelligent Systems in Harsh Environments（克里斯蒂安·多普勒恶劣环境可靠智能系统实验室）
B-XAIC Dataset: Benchmarking Explainable AI for Graph Neural Networks Using Chemical Data
University of Edinburgh（爱丁堡大学）；Jagiellonian University（雅盖隆大学）；Jagiellonian University, Faculty of Mathematics and Computer Science（雅盖隆大学数学与计算机科学学院）；Jagiellonian University, Faculty of Chemistry（雅盖隆大学化学学院）；Ardigen SA（Ardigen公司）
PREF-XAI: Preference-Based Personalized Rule Explanations of Black-Box Machine Learning Models
University of Catania, Department of Economics and Business（卡塔尼亚大学经济与商业系）；Poznan University of Technology, Institute of Computing Science（波兹南技术大学计算科学研究所）；Systems Research Institute, Polish Academy of Sciences（波兰科学院系统研究所）
