Robotic agents need to understand how to interact with objects in their environment, both autonomously and during human-robot interactions. Affordance detection on 3D point clouds, which identifies object regions that allow specific interactions, has traditionally relied on deep learning models like PointNet++, DGCNN, or PointTransformerV3. However, these models operate as black boxes, offering no insight into their decision-making processes. Prototypical Learning methods, such as ProtoPNet, provide an interpretable alternative to black-box models by employing a "this looks like that" case-based reasoning approach. However, they have been primarily applied to image-based tasks. In this work, we apply prototypical learning to models for affordance detection on 3D point clouds. Experiments on the 3D-AffordanceNet benchmark dataset show that prototypical models achieve competitive performance with state-of-the-art black-box models and offer inherent interpretability. This makes prototypical models a promising candidate for human-robot interaction scenarios that require increased trust and safety.
BLaDA: Bridging Language to Functional Dexterous Actions within 3DGS Fields
School of Artificial Intelligence and Robotics, Hunan University（湖南大学人工智能与机器人学院）；National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University（机器人视觉感知与控制技术国家工程研究中心，湖南大学）；Key Laboratory of Advanced Manufacturing Technology of the Ministry of Education, Guizhou University（教育部贵州大学先进制造技术重点实验室）
ProtoPointNet: Prototype-Based Interpretable Classification of 3D Dental Point Clouds with Verifiable Spatial Activations
Adelaide Dental School, Adelaide University（阿德莱德牙科学院，阿德莱德大学）；Australian Institute for Machine Learning, Adelaide University（澳大利亚机器学习研究所，阿德莱德大学）；School of Public Health, Adelaide University（阿德莱德大学公共卫生学院）
University of Edinburgh（爱丁堡大学）；Google Research（谷歌研究院）
Humans excel at acquiring knowledge through observation. For example, we can learn to use new tools by watching demonstrations. This skill is fundamental for intelligent systems to interact with the world. A key step to acquire this skill is to identify what part of the object affords each action, which is called affordance grounding. In this paper, we address this problem and propose a framework called LOCATE that can identify matching object parts across images, to transfer knowledge from images where an object is being used (exocentric images used for learning), to images where the object i…
PiPViT: Patch-based Visual Interpretable Prototypes for Retinal Image Analysis
Christian Doppler Laboratory for Artificial Intelligence in Retina（人工智能在视网膜领域的克里斯蒂安·多普勒实验室）；Institute of Artificial Intelligence（人工智能研究所）；Center for Medical Data Science（医学数据科学中心）；Medical University of Vienna（维也纳医学大学）
We introduce ProtoPool, an interpretable image classification model with a pool of prototypes shared by the classes. The training is more straightforward than in the existing methods because it does not require the pruning stage. It is obtained by introducing a fully differentiable assignment of prototypes to particular classes. Moreover, we introduce a novel focal similarity function to focus the model on the rare foreground features. We show that ProtoPool obtains state-of-the-art accuracy on the CUB-200-2011 and the Stanford Cars datasets, substantially reducing the number of prototypes. We…
Australian Institute for Machine Learning, The University of Adelaide（阿德莱德大学澳大利亚机器学习研究所）；St Vincent’s Institute of Medical Research（圣文森特医学研究所）；St Vincent’s Hospital Melbourne（墨尔本圣文森特医院）
State-of-the-art (SOTA) deep learning mammogram classifiers, trained with weakly-labelled images, often rely on global models that produce predictions with limited interpretability, which is a key barrier to their successful translation into clinical practice. On the other hand, prototype-based models improve interpretability by associating predictions with training image prototypes, but they are less accurate than global models and their prototypes tend to have poor diversity. We address these two issues with the proposal of BRAIxProtoPNet++, which adds interpretability to a global model by e…
University of Twente（特文特大学）；University of Duisburg-Essen（杜伊斯堡-埃森大学）
Prototype-based methods use interpretable representations to address the black-box nature of deep learning models, in contrast to post-hoc explanation methods that only approximate such models. We propose the Neural Prototype Tree (ProtoTree), an intrinsically interpretable deep learning method for fine-grained image recognition. ProtoTree combines prototype learning with decision trees, and thus results in a globally interpretable model by design. Additionally, ProtoTree can locally explain a single prediction by outlining a decision path through the tree. Each node in our binary tree contain…
In this paper, we introduce ProtoPShare, a self-explained method that incorporates the paradigm of prototypical parts to explain its predictions. The main novelty of the ProtoPShare is its ability to efficiently share prototypical parts between the classes thanks to our data-dependent merge-pruning. Moreover, the prototypes are more consistent and the model is more robust to image perturbations than the state of the art method ProtoPNet. We verify our findings on two datasets, the CUB-200-2011 and the Stanford Cars.
Explaining black-box Artificial Intelligence (AI) models is a cornerstone for trustworthy AI and a prerequisite for its use in safety critical applications such that AI models can reliably assist humans in critical decisions. However, instead of trying to explain our models post-hoc, we need models which are interpretable-by-design built on a reasoning process similar to humans that exploits meaningful high-level concepts such as shapes, texture or object parts. Learning such concepts is often hindered by its need for explicit specification and annotation up front. Instead, prototype-based lea…
The interpretability of neural networks has recently received extensive attention. Previous prototype-based explainable networks involved prototype activation in both reasoning and interpretation processes, requiring specific explainable structures for the prototype, thus making the network less accurate as it gains interpretability. Therefore, the decoupling prototypical network (DProtoNet) was proposed to avoid this problem. This new model contains encoder, inference, and interpretation modules. As regards the encoder module, unrestricted feature masks were presented to generate expressive f…
University of Twente（特文特大学）；University of Duisburg-Essen（杜伊斯堡-埃森大学）
Image recognition with prototypes is considered an interpretable alternative for black box deep learning models. Classification depends on the extent to which a test image "looks like" a prototype. However, perceptual similarity for humans can be different from the similarity learned by the classification model. Hence, only visualising prototypes can be insufficient for a user to understand what a prototype exactly represents, and why the model considers a prototype and an image to be similar. We address this ambiguity and argue that prototypes should be explained. We improve interpretability …
In this paper we propose a novel human-centered approach for detecting forgery in face images, using dynamic prototypes as a form of visual explanations. Currently, most state-of-the-art deepfake detections are based on black-box models that process videos frame-by-frame for inference, and few closely examine their temporal inconsistencies. However, the existence of such temporal artifacts within deepfake videos is key in detecting and explaining deepfakes to a supervising human. To this end, we propose Dynamic Prototype Network (DPNet) -- an interpretable and effective solution that utilizes …
CountXplain: Interpretable Cell Counting with Prototype-Based Density Map Estimation
Department of Computer Science, Iowa State University（计算机科学系，爱荷华州立大学）；Department of Genetics, Development, and Cell Biology, Iowa State University（遗传学、发育与细胞生物学系，爱荷华州立大学）；Molecular, Cellular, and Developmental Biology Program, Iowa State University（分子、细胞与发育生物学项目，爱荷华州立大学）；Neuroscience Program, Iowa State University（神经科学项目，爱荷华州立大学）；Nanovaccine Institute, Iowa State University（纳米疫苗研究所，爱荷华州立大学）
Mixture of Gaussian-distributed Prototypes with Generative Modelling for Interpretable and Trustworthy Image Recognition
Australian Institute for Machine Learning (AIML), University of Adelaide（阿德莱德大学澳大利亚机器学习研究所）；Stanford University（斯坦福大学）；Cornell University（康奈尔大学）；Cornell Tech（康奈尔理工学院）；University of Oxford（牛津大学）；St. Vincent’s Institute of Medical Research（圣文森特医学研究所）；Melbourne Integrative Genomics, The University of Melbourne（墨尔本大学墨尔本整合基因组学中心）；St Vincent’s Hospital Melbourne（墨尔本圣文森特医院）；Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey（萨里大学视觉、语音与信号处理中心）；The University of Adelaide（阿德莱德大学）
Hierarchical Vision Transformer with Prototypes for Interpretable Medical Image Classification
Ulm University Medical Center（乌尔姆大学医学中心）；BGZ - Bildgebungszentrum（BGZ影像中心）；i2SouI - Innovative Imaging in Surgical Oncology Ulm（乌尔姆外科肿瘤创新影像中心）；XAIRAD - Cooperation for Artificial Intelligence in Experimental Radiology（XAIRAD实验放射学人工智能合作中心）
Prototype-driven fusion of pathology and spatial transcriptomics for interpretable survival prediction
Department of Biostatistics, MD Anderson Cancer Center（生物统计学系，MD安德森癌症中心）；Department of Translational Molecular Pathology, MD Anderson Cancer Center（转化分子病理学系，MD安德森癌症中心）；The Institute for Data Science in Oncology (IDSO), MD Anderson Cancer Center（肿瘤学数据科学研究所（IDSO），MD安德森癌症中心）
Multimodal Feature Prototype Learning for Interpretable and Discriminative Cancer Survival Prediction
Hangzhou Dianzi University（杭州电子科技大学）；School of Information and Communication Technology, Griffith University（信息与通信技术学院，格里菲斯大学）；Shenzhen Research Institute of Big Data（深圳大数据研究院）；Laboratory of Advanced Theranostic Materials and Technology, Ningbo Institute of Materials Technology and Engineering, Chinese Academy of Sciences（先进诊疗材料与技术实验室，宁波材料技术与工程研究所，中国科学院）；School of Biomedical Engineering, Tsinghua University（生物医学工程学院，清华大学）
Linear discriminant analysis (LDA) is a fundamental method for feature extraction and dimensionality reduction. Despite having many variants, classical LDA has its own importance, as it is a keystone in human knowledge about statistical pattern recognition. For a dataset containing C clusters, the classical solution to LDA extracts at most C-1 features. Here, we introduce a novel solution to classical LDA, called LDA++, that yields C features, each interpretable as measuring similarity to one cluster. This novel solution bridges dimensionality reduction and multiclass classification. Specifica…
Learning Support and Trivial Prototypes for Interpretable Image Classification
Australian Institute for Machine Learning, University of Adelaide（阿德莱德大学澳大利亚机器学习研究所）；Harvard University（哈佛大学）；St Vincent’s Institute of Medical Research（圣文森特医学研究所）；St Vincent’s Hospital Melbourne（墨尔本圣文森特医院）；CVSSP, University of Surrey（萨里大学计算机视觉、语音与信号处理中心）
Proto-FG3D: Prototype-based Interpretable Fine-Grained 3D Shape Classification
School of Information Science and Engineering（信息科学与工程学院）；University of Jinan（济南大学）；School of Control Science and Engineering（控制科学与工程学院）；Shandong University（山东大学）；Department of Computing and Decision Sciences（计算与决策科学系）；Lingnan University（岭南大学）；College of Computer Science（计算机科学学院）；Nankai University（南开大学）
Vision models are interpretable when they classify objects on the basis of features that a person can directly understand. Recently, methods relying on visual feature prototypes have been developed for this purpose. However, in contrast to how humans categorize objects, these approaches have not yet made use of any taxonomical organization of class labels. With such an approach, for instance, we may see why a chimpanzee is classified as a chimpanzee, but not why it was considered to be a primate or even an animal. In this work we introduce a model that uses hierarchically organized prototypes …
DPsurv: Dual-Prototype Evidential Fusion for Uncertainty-Aware and Interpretable Whole-Slide Image Survival Prediction
National University of Singapore；National University of Singapore Guangzhou Research Translation；Innovation Institute；Imperial College London；Peking Union Medical College Hospital, Chinese Academy of Medical Sciences \& Peking Union Medical College；Hunan University；Institute of High Performance Computing, Agency for Science, Technology；Research (A STAR)
Interpretable Medical Image Classification using Prototype Learning and Privileged Information
Experimental Radiology, University Hospital Ulm, Germany（实验放射学，乌尔姆大学医院，德国）；Department of Diagnostic and Interventional Radiology, University Hospital Ulm, Germany（诊断与介入放射学系，乌尔姆大学医院，德国）；i2SouI - Innovative Imaging in Surgical Oncology Ulm, University Hospital Ulm, Germany（i2SouI - 精准影像学在手术肿瘤学中的创新应用，乌尔姆大学医院，德国）
Institute of Biomedical Engineering（生物医学工程研究所）；University of Oxford（牛津大学）
Convolutional neural networks (CNNs) have shown exceptional performance for a range of medical imaging tasks. However, conventional CNNs are not able to explain their reasoning process, therefore limiting their adoption in clinical practice. In this work, we propose an inherently interpretable CNN for regression using similarity-based comparisons (INSightR-Net) and demonstrate our methods on the task of diabetic retinopathy grading. A prototype layer incorporated into the architecture enables visualization of the areas in the image that are most similar to learned prototypes. The final predict…
PIPNet3D: Interpretable Detection of Alzheimer in MRI Scans
University of Pisa（比萨大学）；Fondazione Toscana G Monasterio（托斯卡纳G.莫纳斯特里奥基金会）；University of Marburg（马尔堡大学）；University of Mannheim（曼海姆大学）；University Hospital Marburg（马尔堡大学医院）；Datacation
