We present an interpretable deep model for fine-grained visual recognition. At the core of our method lies the integration of region-based part discovery and attribution within a deep neural network. Our model is trained using image-level object labels, and provides an interpretation of its results via the segmentation of object parts and the identification of their contributions towards classification. To facilitate the learning of object parts without direct supervision, we explore a simple prior of the occurrence of object parts. We demonstrate that this prior, when combined with our region-based part discovery and attribution, leads to an interpretable model that remains highly accurate. Our model is evaluated on major fine-grained recognition datasets, including CUB-200, CelebA and iNaturalist. Our results compare favorably to state-of-the-art methods on classification tasks, and our method outperforms previous approaches on the localization of object parts.
Attributes Grouping and Mining Hashing for Fine-Grained Image Retrieval
Southeast University（东南大学）；Key Laboratory of Measurement and Control of Complex Systems of Engineering, Ministry of Education（教育部复杂工程系统测量与控制重点实验室）；Nanjing Enbo Technology Co., Ltd.（南京恩博科技有限公司）
Beijing University of Posts and Telecommunications（北京邮电大学）；Queen Mary University of London（伦敦玛丽女王大学）；The University of Edinburgh（爱丁堡大学）
In this work we aim to develop a universal sketch grouper. That is, a grouper that can be applied to sketches of any category in any domain to group constituent strokes/segments into semantically meaningful object parts. The first obstacle to this goal is the lack of large-scale datasets with grouping annotation. To overcome this, we contribute the largest sketch perceptual grouping (SPG) dataset to date, consisting of 20,000 unique sketches evenly distributed over 25 object categories. Furthermore, we propose a novel deep universal perceptual grouping model. The model is learned with both gen…
A Simple Interpretable Transformer for Fine-Grained Image Classification and Analysis
The Ohio State University（俄亥俄州立大学）；Amazon Alexa（亚马逊 Alexa）；Virginia Tech（弗吉尼亚理工大学）；Princeton University（普林斯顿大学）；Rensselaer Polytechnic Institute（伦斯勒理工学院）
ConceptHash: Interpretable Fine-Grained Hashing via Concept Discovery
University of Surrey（萨里大学）；iFlyTek-Surrey Joint Research Centre on Artificial Intelligence（讯飞-萨里人工智能联合研究中心）；Surrey Institute for People-Centred Artificial Intelligence（萨里以人为本人工智能研究所）；CVSSP（视觉、语音和信号处理中心）
Grouping has been commonly used in deep metric learning for computing diverse features. However, current methods are prone to overfitting and lack interpretability. In this work, we propose an improved and interpretable grouping method to be integrated flexibly with any metric learning framework. Our method is based on the attention mechanism with a learnable query for each group. The query is fully trainable and can capture group-specific information when combined with the diversity loss. An appealing property of our method is that it naturally lends itself interpretability. The attention sco…
Training an accurate object detector is expensive and time-consuming. One main reason lies in the laborious labeling process, i.e., annotating category and bounding box information for all instances in every image. In this paper, we examine ways to improve performance of deep object detectors without extra labeling. We first explore to group existing categories of high visual and semantic similarities together as one super category (or, a superclass). Then, we study how this knowledge of hierarchical categories can be exploited to better detect object using multi-grained RCNN top branches. Exp…
University of Twente（特文特大学）；University of Duisburg-Essen（杜伊斯堡-埃森大学）
Prototype-based methods use interpretable representations to address the black-box nature of deep learning models, in contrast to post-hoc explanation methods that only approximate such models. We propose the Neural Prototype Tree (ProtoTree), an intrinsically interpretable deep learning method for fine-grained image recognition. ProtoTree combines prototype learning with decision trees, and thus results in a globally interpretable model by design. Additionally, ProtoTree can locally explain a single prediction by outlining a decision path through the tree. Each node in our binary tree contain…
We present ALADIN (All Layer AdaIN); a novel architecture for searching images based on the similarity of their artistic style. Representation learning is critical to visual search, where distance in the learned search embedding reflects image similarity. Learning an embedding that discriminates fine-grained variations in style is hard, due to the difficulty of defining and labelling style. ALADIN takes a weakly supervised approach to learning a representation for fine-grained style similarity of digital artworks, leveraging BAM-FG, a novel large-scale dataset of user generated content groupin…
Transitivity Recovering Decompositions: Interpretable and Robust Fine-Grained Relationships
University of Exeter（埃克塞特大学）；University of Trento（特伦托大学）；University of Tübingen（蒂宾根大学）；MPI for Informatics（马克斯·普朗克信息学研究所）；The Alan Turing Institute（艾伦·图灵研究所）；University of Surrey（萨里大学）
Beihang University（北京航空航天大学）；Shenzhen Academy of Aerospace Technology（深圳航天技术研究院）
Fine-grained visual classification (FGVC) is challenging but more critical than traditional classification tasks. It requires distinguishing different subcategories with the inherently subtle intra-class object variations. Previous works focus on enhancing the feature representation ability using multiple granularities and discriminative regions based on the attention strategy or bounding boxes. However, these methods highly rely on deep neural networks which lack interpretability. We propose an Interpretable Attention Guided Network (IAGN) for fine-grained visual classification. The contribut…
Bosch Center for Artificial Intelligence (BCAI)（博世人工智能中心）；University of Bonn（波恩大学）
To verify and validate networks, it is essential to gain insight into their decisions, limitations as well as possible shortcomings of training data. In this work, we propose a post-hoc, optimization based visual explanation method, which highlights the evidence in the input image for a specific prediction. Our approach is based on a novel technique to defend against adversarial evidence (i.e. faulty evidence due to artefacts) by filtering gradients during optimization. The defense does not depend on human-tuned parameters. It enables explanations which are both fine-grained and preserve the c…
Vision Language Model for Interpretable and Fine-grained Detection of Safety Compliance in Diverse Workplaces
School of Mechanical, Aerospace, and Manufacturing Engineering（机械、航空航天与制造工程学院）；University of Connecticut（康涅狄格大学）；Department of Computer Science（计算机科学系）；University of California Irvine（加州大学欧文分校）
Prompt-CAM: Making Vision Transformers Interpretable for Fine-Grained Analysis
The Ohio State University（俄亥俄州立大学）；University of Tsukuba（筑波大学）；Virginia Tech（弗吉尼亚理工大学）；Princeton University（普林斯顿大学）；Rensselaer Polytechnic Institute（伦斯勒理工学院）
Beihang University（北京航空航天大学）；National University of Defense Technology（国防科技大学）；Princeton University（普林斯顿大学）
A majority of stock 3D models in modern shape repositories are assembled with many fine-grained components. The main cause of such data form is the component-wise modeling process widely practiced by human modelers. These modeling components thus inherently reflect some function-based shape decomposition the artist had in mind during modeling. On the other hand, modeling components represent an over-segmentation since a functional part is usually modeled as a multi-component assembly. Based on these observations, we advocate that labeled segmentation of stock 3D models should not overlook the …
Unveiling Perceptual Artifacts: A Fine-Grained Benchmark for Interpretable AI-Generated Image Detection
Sun Yat-sen University（中山大学）；Peng Cheng Laboratory（鹏城实验室）；Xi’an Jiaotong University（西安交通大学）；Australian National University（澳大利亚国立大学）；Tsinghua University（清华大学）
Proto-FG3D: Prototype-based Interpretable Fine-Grained 3D Shape Classification
School of Information Science and Engineering（信息科学与工程学院）；University of Jinan（济南大学）；School of Control Science and Engineering（控制科学与工程学院）；Shandong University（山东大学）；Department of Computing and Decision Sciences（计算与决策科学系）；Lingnan University（岭南大学）；College of Computer Science（计算机科学学院）；Nankai University（南开大学）
Despite the growing discriminative capabilities of modern deep learning methods for recognition tasks, the inner workings of the state-of-art models still remain mostly black-boxes. In this paper, we propose a systematic interpretation of model parameters and hidden representations of Residual Temporal Convolutional Networks (Res-TCN) for action recognition in time-series data. We also propose a Feature Map Decoder as part of the interpretation analysis, which outputs a representation of model's hidden variables in the same domain as the input. Such analysis empowers us to expose model's chara…
Interpretable Single-View 3D Gaussian Splatting using Unsupervised Hierarchical Disentangled Representation Learning
Shanghai Jiao Tong University（上海交通大学）；Ningbo Institute of Digital Twin, Eastern Institute of Technology（东方理工大学宁波数字孪生研究院）；Hong Kong Polytechnic University（香港理工大学）
Eval3D: Interpretable and Fine-grained Evaluation for 3D Generation
Massachusetts Institute of Technology（麻省理工学院）；University of Washington（华盛顿大学）；New York University（纽约大学）；Wayve（Wayve公司）；Allen Institute for AI（艾伦人工智能研究所）；Cornell University（康奈尔大学）
SurfAge-Net: A Hierarchical Surface-Based Network for Interpretable Fine-Grained Brain Age Prediction
Gansu Provincial Key Laboratory of Wearable Computing, School of Information Science and Engineering, Lanzhou University（甘肃省可穿戴计算重点实验室，信息科学与工程学院，兰州大学）；Department of Medical Imaging Center, Gansu Provincial Maternity and Child-Care Hospital（医学影像中心，甘肃省妇幼保健院）；Department of Magnetic Resonance, Lanzhou University Second Hospital（磁共振科，兰州大学第二医院）；Department of Computer Science and Technology, Harbin Institute of Technology (Shenzhen)（计算机科学与技术系，哈尔滨工业大学（深圳））；Key Laboratory for Biomedical Engineering of Ministry of Education, Department of Biomedical Engineering, College of Biomedical Engineering & Instrument Science, Zhejiang University（教育部生物医学工程重点实验室，生物医学工程学院，浙江大学）
Peking University（北京大学）；Beijing Institute of Technology（北京理工大学）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）；Peng Cheng Laboratory（鹏城实验室）；Pazhou Laboratory (Huangpu)（琶洲实验室（黄埔））
We present a novel two-stage fully sparse convolutional 3D object detection framework, named CAGroup3D. Our proposed method first generates some high-quality 3D proposals by leveraging the class-aware local group strategy on the object surface voxels with the same semantic predictions, which considers semantic consistency and diverse locality abandoned in previous bottom-up approaches. Then, to recover the features of missed voxels due to incorrect voxel-wise segmentation, we build a fully sparse convolutional RoI pooling module to directly aggregate fine-grained spatial information from backb…
Chest radiographs are the most common diagnostic exam in emergency rooms and intensive care units today. Recently, a number of researchers have begun working on large chest X-ray datasets to develop deep learning models for recognition of a handful of coarse finding classes such as opacities, masses and nodules. In this paper, we focus on extracting and learning fine-grained labels for chest X-ray images. Specifically we develop a new method of extracting fine-grained labels from radiology reports by combining vocabulary-driven concept extraction with phrasal grouping in dependency parse trees…
Sun Yat-sen University（中山大学）；SenseTime Group (Limited)（商汤集团有限公司）；CVTE Research（视源研究院）
Instance-level human parsing towards real-world human analysis scenarios is still under-explored due to the absence of sufficient data resources and technical difficulty in parsing multiple instances in a single pass. Several related works all follow the "parsing-by-detection" pipeline that heavily relies on separately trained detection models to localize instances and then performs human parsing for each instance sequentially. Nonetheless, two discrepant optimization targets of detection and parsing lead to suboptimal representation learning and error accumulation for final results. In this w…
Decomposed Vision-Language Alignment for Fine-Grained Open-Vocabulary Segmentation
Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院航空航天信息研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Zhejiang University（浙江大学）
German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）；TU Kaiserslautern（凯泽斯劳滕工业大学）
Classification problems solved with deep neural networks (DNNs) typically rely on a closed world paradigm, and optimize over a single objective (e.g., minimization of the cross-entropy loss). This setup dismisses all kinds of supporting signals that can be used to reinforce the existence or absence of a particular pattern. The increasing need for models that are interpretable by design makes the inclusion of said contextual signals a crucial necessity. To this end, we introduce the notion of Self-Supervised Autogenous Learning (SSAL) models. A SSAL objective is realized through one or more add…
MAProtoNet: A Multi-scale Attentive Interpretable Prototypical Part Network for 3D Magnetic Resonance Imaging Brain Tumor Classification
Tokyo University of Agriculture and Technology (TUAT)（东京农工大学）；RIKEN Center for Advanced Intelligence Project (RIKEN AIP)（理化学研究所先进智能研究中心）；Graduate School of Medicine, Juntendo University（顺天堂大学医学院）；Faculty of Health Data Science, Juntendo University（顺天堂大学健康数据科学学部）；RIKEN（理化学研究所）
We propose an efficient transfer learning method for adapting ImageNet pre-trained Convolutional Neural Network (CNN) to fine-grained image classification task. Conventional transfer learning methods typically face the trade-off between training time and accuracy. By adding "attention module" to each convolutional filters of the pre-trained network, we are able to rank and adjust the importance of each convolutional signal in an end-to-end pipeline. In this report, we show our method can adapt a pre-trianed ResNet50 for a fine-grained transfer learning task within few epochs and achieve accura…
CLIP-AUTT: Test-Time Personalization with Action Unit Prompting for Fine-Grained Video Emotion Recognition
LIVIA, ILLS, Dept. of Systems Engineering, ETS Montreal（LIVIA, ILLS, 系统工程学院, 蒙特利尔高等技术学院）；LIVIA, Dept. of Software and IT Engineering, ETS Montreal（LIVIA, 软件与IT工程学院, 蒙特利尔高等技术学院）；Dept. of Computer Science, École Polytechnique（计算机科学系, 巴黎综合理工学院）
HieroAction: Hierarchically Guided VLM for Fine-Grained Action Analysis
Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing, School of Artificial Intelligence, Beihang University（北京航空航天大学人工智能学院未来区块链与隐私计算高精尖创新中心）；Psyche AI Inc（Psyche AI 公司）；National University of Singapore（新加坡国立大学）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）；MemTensor (Shanghai) Technology Co., Ltd.（MemTensor（上海）科技有限公司）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Jinan University（暨南大学）；Nanjing University of Science and Technology（南京理工大学）；Hefei University of Technology（合肥工业大学）
The University of Adelaide（阿德莱德大学）；Australian Centre for Robotic Vision（澳大利亚机器人视觉中心）
Large-scale datasets have driven the rapid development of deep neural networks for visual recognition. However, annotating a massive dataset is expensive and time-consuming. Web images and their labels are, in comparison, much easier to obtain, but direct training on such automatically harvested images can lead to unsatisfactory performance, because the noisy labels of Web images adversely affect the learned recognition models. To address this drawback we propose an end-to-end weakly-supervised deep learning framework which is robust to the label noise in Web images. The proposed framework rel…
