The rule extraction literature contains the notion of a fidelity-accuracy dilemma: when building an interpretable model of a black box function, optimising for fidelity is likely to reduce performance on the underlying task, and vice versa. I reassert the relevance of this dilemma for the modern field of explainable artificial intelligence, and highlight how it is compounded when the black box is an agent interacting with a dynamic environment. I then discuss two independent research directions - building white box agents and interpreting black box agents - which are both coherent and worthy of attention, but must not be conflated by researchers embarking on projects in the domain of agent interpretability.
One approach for interpreting black-box machine learning models is to find a global approximation of the model using simple interpretable functions, which is called a metamodel (a model of the model). Approximating the black-box with a metamodel can be used to 1) estimate instance-wise feature importance; 2) understand the functional form of the model; 3) analyze feature interactions. In this work, we propose a new method for finding interpretable metamodels. Our approach utilizes Kolmogorov superposition theorem, which expresses multivariate functions as a composition of univariate functions …
University of Milano-Bicocca（米兰比可卡大学）；University of Siena（锡耶纳大学）
The shift from symbolic AI systems to black-box, sub-symbolic, and statistical ones has motivated a rapid increase in the interest toward explainable AI (XAI), i.e. approaches to make black-box AI systems explainable to human decision makers with the aim of making these systems more acceptable and more usable tools and supports. However, we make the point that, rather than always making black boxes transparent, these approaches are at risk of \emph{painting the black boxes white}, thus failing to provide a level of transparency that would increase the system's usability and comprehensibility; …
TriLens: Per-Layer Logit-Lens Entropy for White-Box Hallucination Detection
Binjiang Institute of Zhejiang University（浙江大学滨海学院）；Beijing Normal-Hong Kong Baptist University（北京师范大学-香港 Baptist大学）；Zhejiang University（浙江大学）；GenTel.io；Great Bay University（Great Bay大学）
Moving the Safety Barrier: Dynamic Routing Adaptive Alignment Against White-Box Attacks
Nanjing University of Science and Technology（南京理工大学）；The University of Sydney（悉尼大学）；National University of Singapore（新加坡国立大学）；University of New South Wales（新南威尔士大学）；Nanjing University（南京大学）
Instruction Learning Paradigms: A Dual Perspective on White-box and Black-box LLMs
School of Artificial Intelligence, Beihang University（北京航空航天大学人工智能学院）；Hangzhou International Innovation Institute, Beihang University（北京航空航天大学杭州国际创新研究院）；Nanyang Technological University（南洋理工大学）；University of Leicester（莱斯特大学）；Kuaishou Technology（快手科技）
Saliency strikes back: How filtering out high frequencies improves white-box explanations
Carney Institute for Brain Science, Brown University（布朗大学卡尼脑科学研究所）；CerCo, CNRS（法国国家科学研究中心CerCo研究所）；SNCF（法国国家铁路公司）；Institute of Mathematics of Toulouse, University Paul Sabatier（保罗萨巴蒂埃大学图卢兹数学研究所）
Benchmarking Instance-Centric Counterfactual Algorithms for XAI: From White Box to Black Box
Human Technology Institute, University of Technology Sydney（悉尼科技大学人类技术研究所）；School of Information Systems, Queensland University of Technology（昆士兰科技大学信息系统学院）；INESC-ID Lisboa / Instituto Superior Técnico, ULisboa（INESC-ID里斯本/里斯本大学高等技术学院）
Central South University（中南大学）；National University of Defence Technology（国防科技大学）；Tsinghua University（清华大学）
Hypergraph neural networks (HGNN) have shown superior performance in various deep learning tasks, leveraging the high-order representation ability to formulate complex correlations among data by connecting two or more nodes through hyperedge modeling. Despite the well-studied adversarial attacks on Graph Neural Networks (GNN), there is few study on adversarial attacks against HGNN, which leads to a threat to the safety of HGNN applications. In this paper, we introduce HyperAttack, the first white-box adversarial attack framework against hypergraph neural networks. HyperAttack conducts a white-…
Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院）
Existing works have identified the limitation of top-$1$ attack success rate (ASR) as a metric to evaluate the attack strength but exclusively investigated it in the white-box setting, while our work extends it to a more practical black-box setting: transferable attack. It is widely reported that stronger I-FGSM transfers worse than simple FGSM, leading to a popular belief that transferability is at odds with the white-box attack strength. Our work challenges this belief with empirical finding that stronger attack actually transfers better for the general top-$k$ ASR indicated by the interest …
Glimpse: Enabling White-Box Methods to Use Proprietary Models for Zero-Shot LLM-Generated Text Detection
Zhejiang University（浙江大学）；School of Engineering（工程学院）；Westlake University（西湖大学）；School of Mathematics Physics and Statistics（数学、物理与统计学院）；Shanghai Polytechnic University（上海第二工业大学）；Computer Electrical and Mathematical Science and Engineering Division King Abdullah University of Science and Technology（阿卜杜拉国王科技大学计算机、电气与数学科学与工程学院）
In recent years, deep learning has become prevalent to solve applications from multiple domains. Convolutional Neural Networks (CNNs) particularly have demonstrated state of the art performance for the task of image classification. However, the decisions made by these networks are not transparent and cannot be directly interpreted by a human. Several approaches have been proposed to explain to understand the reasoning behind a prediction made by a network. In this paper, we propose a topology of grouping these methods based on their assumptions and implementations. We focus primarily on white …
Université de Lorraine（洛林大学）；CNRS（法国国家科学研究中心）；Inria（法国国家信息与自动化研究所）；Le Mans Université（勒芒大学）
In the scenario of the Voice Privacy challenge, anonymization is achieved by converting all utterances from a source speaker to match the same target identity; this identity being randomly selected. In this context, an attacker with maximum knowledge about the anonymization system can not infer the target identity. This article proposed to constrain the target selection to a specific identity, i.e., removing the random selection of identity, to evaluate the extreme threat under a whitebox assessment (the attacker has complete knowledge about the system). Targeting a unique identity also allows…
Deep neural networks are widely used in various fields because of their powerful performance. However, recent studies have shown that deep learning models are vulnerable to adversarial attacks, i.e., adding a slight perturbation to the input will make the model obtain wrong results. This is especially dangerous for some systems with high-security requirements, so this paper proposes a new defense method by using the model super-fitting state to improve the model's adversarial robustness (i.e., the accuracy under adversarial attacks). This paper mathematically proves the effectiveness of super-…
The widespread adoption of deep neural networks in computer vision applications has brought forth a significant interest in adversarial robustness. Existing research has shown that maliciously perturbed inputs specifically tailored for a given model (i.e., adversarial examples) can be successfully transferred to another independently trained model to induce prediction errors. Moreover, this property of adversarial examples has been attributed to features derived from predictive patterns in the data distribution. Thus, we are motivated to investigate the following question: Can adversarial defe…
Collab-RAG: Boosting Retrieval-Augmented Generation for Complex Question Answering via White-Box and Black-Box LLM Collaboration
Emory University（埃默里大学）；University of Texas Southwestern Medical Center（德克萨斯大学西南医学中心）；Georgia Institute of Technology（佐治亚理工学院）；SUNY Albany（纽约州立大学奥尔巴尼分校）
Surrogate modeling for interpreting black-box LLMs in medical predictions
Medical Big Data Research Center, Seoul National University Medical Research Center, Seoul National University College of Medicine（首尔国立大学医学院医学大数据研究中心，首尔国立大学医学院）；Department of Biomedical Systems Informatics, Yonsei University College of Medicine（延世大学医学院生物医学系统信息学系）；Laboratory for Computational Physiology, Massachusetts Institute of Technology（麻省理工学院计算生理学实验室）；Division of Pulmonary, Critical Care and Sleep Medicine, Beth Israel Deaconess Medical Center（贝斯以色列德aconess医疗中心呼吸科、重症医学科和睡眠医学科）；Department of Biostatistics, Harvard T.H. Chan School of Public Health（哈佛大学T.H. Chan公共卫生学院生物统计学系）；Department of Cardiology, Yongin Severance Hospital, Yonsei University College of Medicine（延世大学医学院永宁松甫医院心内科）；Center for Digital Health, Yongin Severance Hospital, Yonsei University Health System（延世大学健康系统永宁松甫医院数字健康中心）；Institute for Innovation in Digital Healthcare, Severance Hospital, Seoul, Republic of Korea（首尔松甫医院数字医疗创新研究所）
We present a method for adversarial input generation against black box models for reading comprehension based question answering. Our approach is composed of two steps. First, we approximate a victim black box model via model extraction (Krishna et al., 2020). Second, we use our own white box method to generate input perturbations that cause the approximate model to fail. These perturbed inputs are used against the victim. In experiments we find that our method improves on the efficacy of the AddAny---a white box attack---performed on the approximate model by 25% F1, and the AddSent attack---a…
Université de Bretagne Occidentale - UBO（西布列塔尼大学）；Inserm（法国国家健康与医学研究院）
The advances in machine learning (ML) have greatly improved AI-based diagnosis aid systems in medical imaging. However, being based on collecting medical data specific to individuals induces several security issues, especially in terms of privacy. Even though the owner of the images like a hospital put in place strict privacy protection provisions at the level of its information system, the model trained over his images still holds disclosure potential. The trained model may be accessible to an attacker as: 1) White-box: accessing to the model architecture and parameters; 2) Black box: where h…
