The role of explainability in creating trustworthy artificial intelligence for health care: a comprehensive survey of the terminology, design choices, and evaluation strategies
Erasmus University Medical Center（伊拉斯姆斯大学医学中心）
Aniek F. Markus, Jan A. Kors, Peter R. Rijnbeek
发表信息：
Journal of Biomedical Informatics, 113 (2021), 103655
Artificial intelligence (AI) has huge potential to improve the health and well-being of people, but adoption in clinical practice is still limited. Lack of transparency is identified as one of the main barriers to implementation, as clinicians should be confident the AI system can be trusted. Explainable AI has the potential to overcome this issue and can be a step towards trustworthy AI. In this paper we review the recent literature to provide guidance to researchers and practitioners on the design of explainable AI systems for the health-care domain and contribute to formalization of the field of explainable AI. We argue the reason to demand explainability determines what should be explained as this determines the relative importance of the properties of explainability (i.e. interpretability and fidelity). Based on this, we propose a framework to guide the choice between classes of explainable AI methods (explainable modelling versus post-hoc explanation; model-based, attribution-based, or example-based explanations; global and local explanations). Furthermore, we find that quantitative evaluation metrics, which are important for objective standardized evaluation, are still lacking for some properties (e.g. clarity) and types of explanations (e.g. example-based methods). We conclude that explainable modelling can contribute to trustworthy AI, but the benefits of explainability still need to be proven in practice and complementary measures might be needed to create trustworthy AI in health care (e.g. reporting data quality, performing extensive (external) validation, and regulation).
We are witnessing the emergence of an AI economy and society where AI technologies are increasingly impacting health care, business, transportation and many aspects of everyday life. Many successes have been reported where AI systems even surpassed the accuracy of human experts. However, AI systems may produce errors, can exhibit bias, may be sensitive to noise in the data, and often lack technical and judicial transparency resulting in reduction in trust and challenges in their adoption. These recent shortcomings and concerns have been documented in scientific but also in general press such a…
Trustworthy AI in Digital Health: A Comprehensive Review of Robustness and Explainability
College of Health Solutions, Arizona State University（亚利桑那州立大学健康解决方案学院）；School of Computing and Augmented Intelligence, Arizona State University（亚利桑那州立大学计算与增强智能学院）
Although machine learning (ML) models of AI achieve high performances in medicine, they are not free of errors. Empowering clinicians to identify incorrect model recommendations is crucial for engendering trust in medical AI. Explainable AI (XAI) aims to address this requirement by clarifying AI reasoning to support the end users. Several studies on biomedical imaging achieved promising results recently. Nevertheless, solutions for models using tabular data are not sufficient to meet the requirements of clinicians yet. This paper proposes a methodology to support clinicians in identifying fail…
Towards Bridging the FL Performance-Explainability Trade-Off: A Trustworthy 6G RAN Slicing Use-Case
Iquadrat Informatica（伊夸德拉特信息公司）；Universitat Politècnica de Catalunya (UPC)（加泰罗尼亚理工大学）；i2CAT Foundation（i2CAT基金会）；University of Patras（帕特雷大学）；ISI/ATHENA（雅典娜创新基金会信息研究所）
AI explainability improves the transparency of models, making them more trustworthy. Such goals are motivated by the emergence of deep learning models, which are obscure by nature; even in the domain of images, where deep learning has succeeded the most, explainability is still poorly assessed. In the field of image recognition many feature attribution methods have been proposed with the purpose of explaining a model's behavior using visual cues. However, no metrics have been established so far to assess and select these methods objectively. In this paper we propose a consistent evaluation sco…
To make Explainable AI (XAI) systems trustworthy, understanding harmful effects is just as important as producing well-designed explanations. In this paper, we address an important yet unarticulated type of negative effect in XAI. We introduce explainability pitfalls(EPs), unanticipated negative downstream effects from AI explanations manifesting even when there is no intention to manipulate users. EPs are different from, yet related to, dark patterns, which are intentionally deceptive practices. We articulate the concept of EPs by demarcating it from dark patterns and highlighting the challen…
The recognition or understanding of the scenes observed with a SAR system requires a broader range of cues, beyond the spatial context. These encompass but are not limited to: imaging geometry, imaging mode, properties of the Fourier spectrum of the images or the behavior of the polarimetric signatures. In this paper, we propose a change of paradigm for explainability in data science for the case of Synthetic Aperture Radar (SAR) data to ground the explainable AI for SAR. It aims to use explainable data transformations based on well-established models to generate inputs for AI methods, to prov…
Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））；Peng Cheng Laboratory（鹏城实验室）
Recent works have shown explainability and robustness are two crucial ingredients of trustworthy and reliable text classification. However, previous works usually address one of two aspects: i) how to extract accurate rationales for explainability while being beneficial to prediction; ii) how to make the predictive model robust to different types of adversarial attacks. Intuitively, a model that produces helpful explanations should be more robust against adversarial attacks, because we cannot trust the model that outputs explanations but changes its prediction under small perturbations. To thi…
Pioneer Alpha（先锋阿尔法）；Bangladesh University of Engineering and Technology（孟加拉工程技术大学）；University of Hertfordshire（赫特福德大学）；Westcliff university（韦斯特克利夫大学）；Clark Atlanta University（克拉克亚特兰大大学）
Tencent AI Lab（腾讯人工智能实验室）；Sun Yat-sen University（中山大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Zhejiang University（浙江大学）；University of Texas at Arlington（德克萨斯大学阿灵顿分校）；Shenzhen Research Institute of Big Data（深圳大数据研究院）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Peking University（北京大学）；Tsinghua University（清华大学）；Zhejiang Lab（之江实验室）
Deep graph learning has achieved remarkable progresses in both business and scientific areas ranging from finance and e-commerce, to drug and advanced material discovery. Despite these progresses, how to ensure various deep graph learning algorithms behave in a socially responsible manner and meet regulatory compliance requirements becomes an emerging problem, especially in risk-sensitive domains. Trustworthy graph learning (TwGL) aims to solve the above problems from a technical viewpoint. In contrast to conventional graph learning research which mainly cares about model performance, TwGL con…
Towards Responsible AI Music: an Investigation of Trustworthy Features for Creative Systems
University of Liverpool（利物浦大学）；Sapienza University of Rome（罗马第一大学）；King’s College London（伦敦国王学院）；University of Manchester（曼彻斯特大学）；European Commission’s Joint Research Centre（欧盟委员会联合研究中心）
False Sense of Security in Explainable Artificial Intelligence (XAI)
Institute of Informatics, University of Warsaw（华沙大学信息学研究所）；School of Law, Seoul National University（首尔国立大学法学院）；Dongin Law Group（东仁法律集团）；Wesang Co., Ltd.（Wesang有限公司）；Informatism LLC（Informatism有限责任公司）
University of Massachusetts Lowell（马萨诸塞大学洛厄尔分校）；IQVIA（艾昆纬）
Veteran mental health is a significant national problem as large number of veterans are returning from the recent war in Iraq and continued military presence in Afghanistan. While significant existing works have investigated twitter posts-based Post Traumatic Stress Disorder (PTSD) assessment using blackbox machine learning techniques, these frameworks cannot be trusted by the clinicians due to the lack of clinical explainability. To obtain the trust of clinicians, we explore the big question, can twitter posts provide enough information to fill up clinical PTSD assessment surveys that have be…
Policy-Driven AI in Dataspaces: Taxonomy, Explainability, and Pathways for Compliant Innovation
Department of CST Tsinghua University Beijing, China（计算机科学与技术系 清华大学 北京中国）；Department of CSE Chandigarh University Mohali, India（计算机科学与工程系 印度昌迪加尔大学 摩哈利）
Position: Multi-Agent Algorithmic Care Systems Demand Contestability for Trustworthy AI
Analytics Everywhere Lab, University of New Brunswick, Canada（安利克斯 everywhere 实验室，新不伦瑞克大学，加拿大）；National Research Council Canada, Canada（加拿大国家研究委员会，加拿大）；Thompson Rivers University, Canada（汤普森河大学，加拿大）；ISB Corporation, Japan（ISB公司，日本）；University of Northern British Columbia, Canada（北北加拿大大学，加拿大）
On the Black-box Explainability of Object Detection Models for Safe and Trustworthy Industrial Applications
TECNALIA（泰克纳利亚技术中心）；Basque Research and Technology Alliance (BRTA)（巴斯克研究与技术联盟）；University of Deusto（德乌斯托大学）；University of the Basque Country (UPV/EHU)（巴斯克大学）
Concerns about the societal impact of AI-based services and systems has encouraged governments and other organisations around the world to propose AI policy frameworks to address fairness, accountability, transparency and related topics. To achieve the objectives of these frameworks, the data and software engineers who build machine-learning systems require knowledge about a variety of relevant supporting tools and techniques. In this paper we provide an overview of technologies that support building trustworthy machine learning systems, i.e., systems whose properties justify that people place…
The currently dominating artificial intelligence and machine learning technology, neural networks, builds on inductive statistical learning. Neural networks of today are information processing systems void of understanding and reasoning capabilities, consequently, they cannot explain promoted decisions in a humanly valid form. In this work, we revisit and use fundamental philosophy of science theories as an analytical lens with the goal of revealing, what can be expected, and more importantly, not expected, from methods that aim to explain decisions promoted by a neural network. By conducting …
University of Washington（华盛顿大学）；Arthur AI（亚瑟人工智能公司）
Machine learning plays a role in many deployed decision systems, often in ways that are difficult or impossible to understand by human stakeholders. Explaining, in a human-understandable way, the relationship between the input and output of machine learning models is essential to the development of trustworthy machine learning based systems. A burgeoning body of research seeks to define the goals and methods of explainability in machine learning. In this paper, we seek to review and categorize research on counterfactual explanations, a specific class of explanation that provides a link between…
Michigan State University（密歇根州立大学）；The Hong Kong Polytechnic University（香港理工大学）；Twitter（推特）；Tsinghua University（清华大学）
In the past few decades, artificial intelligence (AI) technology has experienced swift developments, changing everyone's daily life and profoundly altering the course of human society. The intention of developing AI is to benefit humans, by reducing human labor, bringing everyday convenience to human lives, and promoting social good. However, recent research and AI applications show that AI can cause unintentional harm to humans, such as making unreliable decisions in safety-critical scenarios or undermining fairness by inadvertently discriminating against one group. Thus, trustworthy AI has a…
Trustworthy and Explainable Deep Reinforcement Learning for Safe and Energy-Efficient Process Control: A Use Case in Industrial Compressed Air Systems
Fraunhofer Institute for Manufacturing Engineering and Automation IPA（弗劳恩霍夫制造工程与自动化研究所）；Institute for Energy Efficiency in Production EEP, University of Stuttgart（生产能效研究所，斯图加特大学）；Institute of Industrial Manufacturing and Management IFF, University of Stuttgart（工业制造与管理研究所，斯图加特大学）
Istituto di Matematica Applicata e Tecnologie Informatiche "Enrico Magenes", Consiglio Nazionale delle Ricerche（意大利国家研究委员会“恩里科·马格内斯”应用数学与信息技术研究所）；Institute of Information Science and Technologies "A. Faedo" of the National Research Council of Italy（意大利国家研究委员会“A.法埃多”信息科学与技术研究所）；Poznań University of Technology（波兹南理工大学）
Reproducibility is one of the core dimensions that concur to deliver Trustworthy Artificial Intelligence. Broadly speaking, reproducibility can be defined as the possibility to reproduce the same or a similar experiment or method, thereby obtaining the same or similar results as the original scientists. It is an essential ingredient of the scientific method and crucial for gaining trust in relevant claims. A reproducibility crisis has been recently acknowledged by scientists and this seems to affect even more Artificial Intelligence and Machine Learning, due to the complexity of the models at …
Brown University（布朗大学）；Université de Toulouse（图卢兹大学）；IRT Saint-Exupery（圣埃克苏佩里技术研究所）
A plethora of methods have been proposed to explain how deep neural networks reach their decisions but comparatively, little effort has been made to ensure that the explanations produced by these methods are objectively relevant. While several desirable properties for trustworthy explanations have been formulated, objective measures have been harder to derive. Here, we propose two new measures to evaluate explanations borrowed from the field of algorithmic stability: mean generalizability MeGe and relative consistency ReCo. We conduct extensive experiments on different network architectures, c…
A Proposed S.C.O.R.E. Evaluation Framework for Large Language Models : Safety, Consensus, Objectivity, Reproducibility and Explainability
Singapore National Eye Centre（新加坡全国眼科中心）；Singapore Eye Research Institute（新加坡眼科研究所）；Singapore Health Services（新加坡保健服务集团）；Singapore General Hospital（新加坡中央医院）；University of California San Francisco（加州大学旧金山分校）；Duke-NUS Medical School（杜克-新加坡国立大学医学院）；Stanford University（斯坦福大学）；Nanyang Technological University（南洋理工大学）；Tsinghua University（清华大学）；Sun Yat Sen University（中山大学）
DarwinAI Corp.（DarwinAI公司）；University of Waterloo（滑铁卢大学）；Waterloo Artificial Intelligence Institute（滑铁卢人工智能研究所）
The COVID-19 pandemic continues to have a devastating global impact, and has placed a tremendous burden on struggling healthcare systems around the world. Given the limited resources, accurate patient triaging and care planning is critical in the fight against COVID-19, and one crucial task within care planning is determining if a patient should be admitted to a hospital's intensive care unit (ICU). Motivated by the need for transparent and trustworthy ICU admission clinical decision support, we introduce COVID-Net Clinical ICU, a neural network for ICU admission prediction based on patient cl…
The Hong Kong Polytechnic University（香港理工大学）；City University of Hong Kong（香港城市大学）；Michigan State University（密歇根州立大学）；The Hong Kong University of Science and Technology（香港科技大学）
As one of the most successful AI-powered applications, recommender systems aim to help people make appropriate decisions in an effective and efficient way, by providing personalized suggestions in many aspects of our lives, especially for various human-oriented online services such as e-commerce platforms and social media sites. In the past few decades, the rapid developments of recommender systems have significantly benefited human by creating economic value, saving time and effort, and promoting social good. However, recent studies have found that data-driven recommender systems can pose ser…
