Interpreting individual neurons in deep neural networks is a crucial step towards understanding their complex decision-making processes and ensuring AI safety. Despite recent progress in neuron labeling, existing methods often limit the search space to predefined concept vocabularies or produce overly specific descriptions that fail to capture higher-order, global concepts. We introduce LINE, a novel, training-free iterative approach tailored for open-vocabulary concept labeling in vision models. Operating in a strictly black-box setting, LINE leverages a large language model and a text-to-image generator to iteratively propose and refine concepts in a closed loop, guided by activation history. LINE achieves state-of-the-art performance across multiple model architectures, yielding AUC improvements of up to 0.11 on ImageNet and 0.05 on Places365, while discovering, on average, 27% of new concepts missed by predefined vocabularies. Beyond identifying the top concept, LINE provides a complete generation history, enabling polysemanticity evaluation and producing visual explanations that rival gradient-dependent activation maximization methods. The source code will be made available soon.
Georgia Institute of Technology（佐治亚理工学院）；Facebook AI Research（Facebook人工智能研究院）
We propose a technique for producing "visual explanations" for decisions from a large class of CNN-based models, making them more transparent. Our approach - Gradient-weighted Class Activation Mapping (Grad-CAM), uses the gradients of any target concept, flowing into the final convolutional layer to produce a coarse localization map highlighting important regions in the image for predicting the concept. Grad-CAM is applicable to a wide variety of CNN model-families: (1) CNNs with fully-connected layers, (2) CNNs used for structured outputs, (3) CNNs used in tasks with multimodal inputs or rein…
Exploring Iterative Enhancement for Improving Learnersourced Multiple-Choice Question Explanations with Large Language Models
Xtracta（埃克斯特拉克塔公司）；School of Computer Science, University of Auckland（奥克兰大学计算机科学学院）；School of Computer Science and Engineering, Sun Yat-Sen University（中山大学计算机科学与工程学院）；School of Life and Environmental Sciences, University of Sydney（悉尼大学生命与环境科学学院）
FaithLM: Towards Faithful Explanations for Large Language Models
Rice University（里士大学）；Texas A&M University（德克萨斯农工大学）；Wake Forest University（沃杰森林大学）；New Jersey Institute of Technology（新泽西理工学院）；John Hopkins University（约翰霍普金斯大学）；Google Research（谷歌研究）；Meta Platforms, Inc.（元宇宙平台公司）
We describe a procedure for explaining neurons in deep representations by identifying compositional logical concepts that closely approximate neuron behavior. Compared to prior work that uses atomic labels as explanations, analyzing neurons compositionally allows us to more precisely and expressively characterize their behavior. We use this procedure to answer several questions on interpretability in models for vision and natural language processing. First, we examine the kinds of abstractions learned by neurons. In image classification, we find that many neurons learn highly abstract but sema…
Beihang University（北京航空航天大学）；Peng Cheng Laboratory（鹏城实验室）；RIKEN AIP（理化学研究所人工智能研究中心）；University of Tokyo（东京大学）；Ningbo Institute of Industrial Technology, CAS（中国科学院宁波工业技术研究院）
Though deep learning has shown successful performance in classifying the label and severity stage of certain diseases, most of them give few explanations on how to make predictions. Inspired by Koch's Postulates, the foundation in evidence-based medicine (EBM) to identify the pathogen, we propose to exploit the interpretability of deep learning application in medical diagnosis. By determining and isolating the neuron activation patterns on which diabetic retinopathy (DR) detector relies to make decisions, we demonstrate the direct relation between the isolated neuron activation and lesions for…
WASD: Locating Critical Neurons as Sufficient Conditions for Explaining and Controlling LLM Behavior
Key Lab of High Confidence Software Technologies (Peking University), Ministry of Education（高可信软件技术教育部重点实验室（北京大学））；School of Computer Science, Peking University（北京大学计算机学院）
Neuronal Group Communication for Efficient Neural representation
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；School of Computer Science and Technology, University of Chinese Academy of Sciences（中国科学院大学计算机科学与技术学院）
本文提出 NGC 框架，将网络建模为神经元群组通过低维信号迭代交互的动力系统，以低秩模块化表示压缩 LLM，并借助稳定性指标解释推理涌现，实验显示其优于同类低秩方法。
School of Engineering and Built Environment, Griffith University（格里菲斯大学工程与建筑环境学院）；Institute for Integrated and Intelligent Systems, Griffith University（格里菲斯大学集成与智能系统研究所）；School of Electronic Information and Artificial Intelligence, Shaanxi University of Science and Technology（陕西科技大学电子信息与人工智能学院）
Geometric Intelligence（几何智能）；Uber AI Labs（优步人工智能实验室）；University of Wyoming（怀俄明大学）；Montreal Institute for Learning Algorithms（蒙特利尔学习算法研究所）；University of Freiburg（弗莱堡大学）
Generating high-resolution, photo-realistic images has been a long-standing goal in machine learning. Recently, Nguyen et al. (2016) showed one interesting way to synthesize novel images by performing gradient ascent in the latent space of a generator network to maximize the activations of one or multiple neurons in a separate classifier network. In this paper we extend this method by introducing an additional prior on the latent code, improving both sample quality and sample diversity, leading to a state-of-the-art generative model that produces high quality images at higher resolutions (227x…
Contrastive Semantic Projection: Faithful Neuron Labeling with Contrastive Examples
Fraunhofer Heinrich Hertz Institute（弗劳恩霍夫海因里希·赫兹研究所）；Technische Universität Berlin（柏林技术大学）；BIFOLD – Berlin Institute for the Foundations of Learning and Data（柏林学习与数据基础研究所）；Centre of eXplainable Artificial Intelligence, Technological University Dublin（可解释人工智能中心，都柏林技术大学）
Deep neural networks have achieved great success in many real-world applications, yet it remains unclear and difficult to explain their decision-making process to an end-user. In this paper, we address the explainable AI problem for deep neural networks with our proposed framework, named IASSA, which generates an importance map indicating how salient each pixel is for the model's prediction with an iterative and adaptive sampling module. We employ an affinity matrix calculated on multi-level deep learning features to explore long-range pixel-to-pixel correlation, which can shift the saliency v…
Georgia Institute of Technology（佐治亚理工学院）；Facebook（脸书）
Individual neurons in convolutional neural networks supervised for image-level classification tasks have been shown to implicitly learn semantically meaningful concepts ranging from simple textures and shapes to whole or partial objects - forming a "dictionary" of concepts acquired through the learning process. In this work we introduce a simple, efficient zero-shot learning approach based on this observation. Our approach, which we call Neuron Importance-AwareWeight Transfer (NIWT), learns to map domain knowledge about novel "unseen" classes onto this dictionary of learned concepts and then o…
State-of-the-art neural networks are getting deeper and wider. While their performance increases with the increasing number of layers and neurons, it is crucial to design an efficient deep architecture in order to reduce computational and memory costs. Designing an efficient neural network, however, is labor intensive requiring many experiments, and fine-tunings. In this paper, we introduce network trimming which iteratively optimizes the network by pruning unimportant neurons based on analysis of their outputs on a large dataset. Our algorithm is inspired by an observation that the outputs of…
Deep learning is increasingly used in decision-making tasks. However, understanding how neural networks produce final predictions remains a fundamental challenge. Existing work on interpreting neural network predictions for images often focuses on explaining predictions for single images or neurons. As predictions are often computed from millions of weights that are optimized over millions of images, such explanations can easily miss a bigger picture. We present Summit, an interactive system that scalably and systematically summarizes and visualizes what features a deep learning model has lear…
