We share observations and challenges from an ongoing effort to implement Explainable AI (XAI) in a domain-specific workflow for cybersecurity analysts. Specifically, we briefly describe a preliminary case study on the use of XAI for source code classification, where accurate assessment and timeliness are paramount. We find that the outputs of state-of-the-art saliency explanation techniques (e.g., SHAP or LIME) are lost in translation when interpreted by people with little AI expertise, despite these techniques being marketed for non-technical users. Moreover, we find that popular XAI techniques offer fewer insights for real-time human-AI workflows when they are post hoc and too localized in their explanations. Instead, we observe that cyber analysts need higher-level, easy-to-digest explanations that can offer as little disruption as possible to their workflows. We outline unaddressed gaps in practical and effective XAI, then touch on how emerging technologies like Large Language Models (LLMs) could mitigate these existing obstacles.
Cybersecurity vendors consistently apply AI (Artificial Intelligence) to their solutions and many cybersecurity domains can benefit from AI technology. However, black-box AI techniques present some difficulties in comprehension and adoption by its operators, given that their decisions are not always humanly understandable (as is usually the case with deep neural networks, for example). Since it aims to make the operation of AI algorithms more interpretable for its users and developers, XAI (eXplainable Artificial Intelligence) can be used to address this issue. Through a systematic literature …
University of Missouri（密苏里大学）；Kennesaw State University（肯尼索州立大学）；University of Texas at San Antonio（德克萨斯大学圣安东尼奥分校）
Cybersickness is a common ailment associated with virtual reality (VR) user experiences. Several automated methods exist based on machine learning (ML) and deep learning (DL) to detect cybersickness. However, most of these cybersickness detection methods are perceived as computationally intensive and black-box methods. Thus, those techniques are neither trustworthy nor practical for deploying on standalone energy-constrained VR head-mounted devices (HMDs). In this work, we present an explainable artificial intelligence (XAI)-based framework, LiteVR, for cybersickness detection, explaining the …
A plethora of recent research has proposed several automated methods based on machine learning (ML) and deep learning (DL) to detect cybersickness in Virtual reality (VR). However, these detection methods are perceived as computationally intensive and black-box methods. Thus, those techniques are neither trustworthy nor practical for deploying on standalone VR head-mounted displays (HMDs). This work presents an explainable artificial intelligence (XAI)-based framework VR-LENS for developing cybersickness detection ML models, explaining them, reducing their size, and deploying them in a Qualcom…
US Army Futures Command（美国陆军未来司令部）；US Army Engineer Research and Development Center（美国陆军工程研究与发展中心）；University of Massachusetts（马萨诸塞大学）
To benefit from AI advances, users and operators of AI systems must have reason to trust it. Trust arises from multiple interactions, where predictable and desirable behavior is reinforced over time. Providing the system's users with some understanding of AI operations can support predictability, but forcing AI to explain itself risks constraining AI capabilities to only those reconcilable with human cognition. We argue that AI systems should be designed with features that build trust by bringing decision-analytic perspectives and formal tools into AI. Instead of trying to achieve explainable …
A Survey on Explainable Artificial Intelligence for Cybersecurity
Princess Sumaya University for Technology（苏玛雅公主科技大学）；Concordia University（康考迪亚大学）；Polytechnique Montréal（蒙特利尔理工学院）；Khalifa University（哈利法大学）；University of Waterloo（滑铁卢大学）；Lebanese American University（黎巴嫩美国大学）；New York University（纽约大学）
Cybersecurity is a domain where the data distribution is constantly changing with attackers exploring newer patterns to attack cyber infrastructure. Intrusion detection system is one of the important layers in cyber safety in today's world. Machine learning based network intrusion detection systems started showing effective results in recent years. With deep learning models, detection rates of network intrusion detection system are improved. More accurate the model, more the complexity and hence less the interpretability. Deep neural networks are complex and hard to interpret which makes diffi…
Explainable AI-Driven Cyber Risk Analytics and Model Reliability Assessment for Intelligent Governance of U.S. Critical Infrastructure: An XGBoost and SHAP-Based Intrusion Detection Framework
Department of Business Information Systems, Central Michigan University（中央密歇根大学商业信息系统系）；Department of Information Studies, Trine University（特林大学信息学系）；Department of Computer Science, Maharishi International University（Maharishi国际大学计算机科学系）
IBM Research AI（IBM研究院AI）；IBM Design（IBM设计）；IBM Research（IBM研究院）
A pervasive design issue of AI systems is their explainability--how to provide appropriate information to help users understand the AI. The technical field of explainable AI (XAI) has produced a rich toolbox of techniques. Designers are now tasked with the challenges of how to select the most suitable XAI techniques and translate them into UX solutions. Informed by our previous work studying design challenges around XAI UX, this work proposes a design process to tackle these challenges. We review our and related prior work to identify requirements that the process should fulfill, and according…
Explainable AI-based Intrusion Detection System for Industry 5.0: An Overview of the Literature, associated Challenges, the existing Solutions, and Potential Research Directions
Hamad bin Khalifa University（哈马德·本·哈利法大学）；Qatar Computing Research Institute（卡塔尔计算研究所）；Munster Technological University（芒斯特理工大学）；Toronto Metropolitan University（多伦多都会大学）
LENS-XAI: Redefining Lightweight and Explainable Network Security through Knowledge Distillation and Variational Autoencoders for Scalable Intrusion Detection in Cybersecurity
Kırıkkale University（克尔克卡拉大学）；University College Dublin（都柏林大学学院）；CeADAR；School of Computer Science（计算机科学学院）
XAI-CF -- Examining the Role of Explainable Artificial Intelligence in Cyber Forensics
Department of Information Security, College of Computer Science and Engineering, University of Ha’il, Ha’il, Saudi Arabia（信息安全系，计算机科学与工程学院，哈伊尔大学，沙特阿拉伯）
Arizona State University（亚利桑那州立大学）；Institute for Computer Science and Engineering (ICIC UNS–CONICET)（计算机科学与工程研究所（ICIC UNS–CONICET））；Institute for Computer Science Research (ICC UBA–CONICET)（计算机科学研究所（ICC UBA–CONICET））；Universidad Nacional del Sur（南方国立大学）；Universidad Nacional de Entre Ríos（恩特雷里奥斯国立大学）；Universidad de Buenos Aires（布宜诺斯艾利斯大学）
With the availability of large datasets and ever-increasing computing power, there has been a growing use of data-driven artificial intelligence systems, which have shown their potential for successful application in diverse areas. However, many of these systems are not able to provide information about the rationale behind their decisions to their users. Lack of understanding of such decisions can be a major drawback, especially in critical domains such as those related to cybersecurity. In light of this problem, in this paper we make three contributions: (i) proposal and discussion of deside…
In this short paper, we argue for a refocusing of XAI around human learning goals. Drawing upon approaches and theories from the learning sciences, we propose a framework for the learner-centered design and evaluation of XAI systems. We illustrate our framework through an ongoing case study in the context of AI-augmented social work.
Exploring Energy Landscapes for Minimal Counterfactual Explanations: Applications in Cybersecurity and Beyond
Netcompany-Intrasoft S.A.（Netcompany-Intrasoft公司）；Hellenic Mediterranean University（希腊地中海大学）；Harokopio University of Athens（雅典哈罗科皮奥大学）；University of Western Macedonia（西马其顿大学）
Mississippi State University（密西西比州立大学）；U.S Army Engineer Research and Development Center（美国陆军工程研究与发展中心）
The application of Artificial Intelligence (AI) and Machine Learning (ML) to cybersecurity challenges has gained traction in industry and academia, partially as a result of widespread malware attacks on critical systems such as cloud infrastructures and government institutions. Intrusion Detection Systems (IDS), using some forms of AI, have received widespread adoption due to their ability to handle vast amounts of data with a high prediction accuracy. These systems are hosted in the organizational Cyber Security Operation Center (CSoC) as a defense tool to monitor and detect malicious network…
Brandon University（布兰登大学）；Pandit Deendayal Energy University（潘迪特·丁达亚尔能源大学）；Vellore Institute of Technology（韦洛尔理工学院）；Symbiosis International Deemed University（共生国际被视为大学）；The Open University（开放大学）
In the past few years, artificial intelligence (AI) techniques have been implemented in almost all verticals of human life. However, the results generated from the AI models often lag explainability. AI models often appear as a blackbox wherein developers are unable to explain or trace back the reasoning behind a specific decision. Explainable AI (XAI) is a rapid growing field of research which helps to extract information and also visualize the results generated with an optimum transparency. The present study provides and extensive review of the use of XAI in cybersecurity. Cybersecurity enab…
University College Dublin（都柏林大学学院）；Technological University Dublin（都柏林理工大学）；Ho Chi Minh City University of Technology and Education（胡志明市工业大学）；Vellore Institute of Technology（韦洛尔理工学院）；Lebanese American University（黎巴嫩美国大学）
When 5G began its commercialisation journey around 2020, the discussion on the vision of 6G also surfaced. Researchers expect 6G to have higher bandwidth, coverage, reliability, energy efficiency, lower latency, and an integrated "human-centric" network system powered by artificial intelligence (AI). Such a 6G network will lead to an excessive number of automated decisions made in real-time. These decisions can range widely, from network resource allocation to collision avoidance for self-driving cars. However, the risk of losing control over decision-making may increase due to high-speed, dat…
Human-Centered Explainable AI for Security Enhancement: A Deep Intrusion Detection Framework
Department of Computer Science and Engineering, United International University (UIU), Dhaka 1212, Bangladesh（计算机科学与工程系，国际联合大学（UIU），达卡1212，孟加拉国）；Department of Electrical and Electronic Engineering, Islamic University of Technology, Gazipur 1704, Bangladesh（电气与电子工程系，伊斯兰科技大学，加兹ipur 1704，孟加拉国）；Department of Computer Science and Engineering (CSE), University of Asia Pacific (UAP), Dhaka 1207, Bangladesh（计算机科学与工程系（CSE），亚洲太平洋大学（UAP），达卡1207，孟加拉国）；Department of Information Systems, University of Maryland, Baltimore, 21250, Maryland, USA（信息系统系，马里兰大学，巴尔的摩，21250，美国）；College Of Information Sciences and Technology, Pennsylvania State University, University Park, PA 16802, USA（信息科学与技术学院，宾夕法尼亚州立大学，大学公园，PA 16802，美国）；Department of Information Technology, Washington University of Science and Technology, Alexandria, VA（信息技术系，科学与技术华盛顿大学，亚历山大，VA）；College of Engineering and Information Technology, University of Maryland, College Park, 20742, Maryland, USA（工程与信息技术学院，马里兰大学，学院公园，20742，美国）
Conversational versus Dashboard Explainable AI for UAV Intrusion Detection: An Empirical Study of Operator Trust and Reliance
Phenikaa University（菲卡大学）；Phenikaa School of Computing（菲卡计算机学院）；MobiFone Corporation（MobiFone集团）；MobiFone HighTech Center（MobiFone高科技中心）；National Economics University（国民经济大学）；College of Technology（技术学院）
Detecting AI-Generated Text in Educational Content: Leveraging Machine Learning and Explainable AI for Academic Integrity
Arab American University（阿拉伯美国大学）；Fu Foundation School of Engineering and Applied Science（傅 Foundation 工程与应用科学学院）；Columbia University（哥伦比亚大学）；Eastern Michigan University（东密歇根大学）；Texas A&M University（德克萨斯农工大学）
To understand how end-users conceptualize consequences of cyber security attacks, we performed a card sorting study, a well-known technique in Cognitive Sciences, where participants were free to group the given consequences of chosen cyber attacks into as many categories as they wished using rationales they see fit. The results of the open card sorting study showed a large amount of inter-participant variation making the research team wonder how the consequences of security attacks were comprehended by the participants. As an exploration of whether it is possible to explain user's mental model…
Cybersecurity analysts work on large communication data sets to perform investigative analysis by painstakingly going over thousands of email conversations to find potential scamming activities and the network of cyber scammers. Traditionally,experts used email clients, database systems and text editors to perform this investigation. With the advent of technology,elaborate tools that summarize data more efficiently by using cutting edge data visualization techniques have come out. Beagle[1] is one such tool which visualizes the large communication data using different panels such that the insp…
Explainable AI (XAI) research has been booming, but the question "$\textbf{To whom}$ are we making AI explainable?" is yet to gain sufficient attention. Not much of XAI is comprehensible to non-AI experts, who nonetheless, are the primary audience and major stakeholders of deployed AI systems in practice. The gap is glaring: what is considered "explained" to AI-experts versus non-experts are very different in practical scenarios. Hence, this gap produced two distinct cultures of expectations, goals, and forms of XAI in real-life AI deployments. We advocate that it is critical to develop XAI me…
