Fine-grained visual classification (FGVC) is challenging but more critical than traditional classification tasks. It requires distinguishing different subcategories with the inherently subtle intra-class object variations. Previous works focus on enhancing the feature representation ability using multiple granularities and discriminative regions based on the attention strategy or bounding boxes. However, these methods highly rely on deep neural networks which lack interpretability. We propose an Interpretable Attention Guided Network (IAGN) for fine-grained visual classification. The contributions of our method include: i) an attention guided framework which can guide the network to extract discriminitive regions in an interpretable way; ii) a progressive training mechanism obtained to distill knowledge stage by stage to fuse features of various granularities; iii) the first interpretable FGVC method with a competitive performance on several standard FGVC benchmark datasets.
Technical University of Munich（慕尼黑工业大学）；Sapienza University of Rome（罗马第一大学）；Magic Leap（Magic Leap公司）；Google（谷歌）
Few-shot fine-grained classification and person search appear as distinct tasks and literature has treated them separately. But a closer look unveils important similarities: both tasks target categories that can only be discriminated by specific object details; and the relevant models should generalize to new categories, not seen during training. We propose a novel unified Query-Guided Network (QGN) applicable to both tasks. QGN consists of a Query-guided Siamese-Squeeze-and-Excitation subnetwork which re-weights both the query and gallery features across all network layers, a Query-guided Reg…
A key challenge in fine-grained recognition is how to find and represent discriminative local regions. Recent attention models are capable of learning discriminative region localizers only from category labels with reinforcement learning. However, not utilizing any explicit part information, they are not able to accurately find multiple distinctive regions. In this work, we introduce an attribute-guided attention localization scheme where the local region localizers are learned under the guidance of part attribute descriptions. By designing a novel reward strategy, we are able to learn to loca…
Boston University（波士顿大学）；Istituto Italiano di Tecnologia（意大利技术研究院）；Adobe Research（奥多比研究中心）；University of Verona（维罗纳大学）
We propose Guided Zoom, an approach that utilizes spatial grounding of a model's decision to make more informed predictions. It does so by making sure the model has "the right reasons" for a prediction, defined as reasons that are coherent with those used to make similar correct decisions at training time. The reason/evidence upon which a deep convolutional neural network makes a prediction is defined to be the spatial grounding, in the pixel space, for a specific class conditional probability in the model output. Guided Zoom examines how reasonable such evidence is for each of the top-k predi…
In this work, we present a novel mask guided attention (MGA) method for fine-grained patchy image classification. The key challenge of fine-grained patchy image classification lies in two folds, ultra-fine-grained inter-category variances among objects and very few data available for training. This motivates us to consider employing more useful supervision signal to train a discriminative model within limited training samples. Specifically, the proposed MGA integrates a pre-trained semantic segmentation model that produces auxiliary supervision signal, i.e., patchy attention mask, enabling a d…
GUIDED: Granular Understanding via Identification, Detection, and Discrimination for Fine-Grained Open-Vocabulary Object Detection
School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机科学与工程学院）；Meituan（美团）；GuangDong Province Key Laboratory of Information Security Technology（广东省信息安全技术重点实验室）；Research Institute, Sun Yat-sen University（中山大学深圳研究院）
Due to the rapid temporal and fine-grained nature of complex human assembly atomic actions, traditional action segmentation approaches requiring the spatial (and often temporal) down sampling of video frames often loose vital fine-grained spatial and temporal information required for accurate classification within the manufacturing domain. In order to fully utilise higher resolution video data (often collected within the manufacturing domain) and facilitate real time accurate action segmentation - required for human robot collaboration - we present a novel hand location guided high resolution …
Zero-Shot Learning (ZSL) is achieved via aligning the semantic relationships between the global image feature vector and the corresponding class semantic descriptions. However, using the global features to represent fine-grained images may lead to sub-optimal results since they neglect the discriminative differences of local regions. Besides, different regions contain distinct discriminative information. The important regions should contribute more to the prediction. To this end, we propose a novel stacked semantics-guided attention (S2GA) model to obtain semantic relevant features by using in…
Unveiling Perceptual Artifacts: A Fine-Grained Benchmark for Interpretable AI-Generated Image Detection
Sun Yat-sen University（中山大学）；Peng Cheng Laboratory（鹏城实验室）；Xi’an Jiaotong University（西安交通大学）；Australian National University（澳大利亚国立大学）；Tsinghua University（清华大学）
Proto-FG3D: Prototype-based Interpretable Fine-Grained 3D Shape Classification
School of Information Science and Engineering（信息科学与工程学院）；University of Jinan（济南大学）；School of Control Science and Engineering（控制科学与工程学院）；Shandong University（山东大学）；Department of Computing and Decision Sciences（计算与决策科学系）；Lingnan University（岭南大学）；College of Computer Science（计算机科学学院）；Nankai University（南开大学）
Despite the growing discriminative capabilities of modern deep learning methods for recognition tasks, the inner workings of the state-of-art models still remain mostly black-boxes. In this paper, we propose a systematic interpretation of model parameters and hidden representations of Residual Temporal Convolutional Networks (Res-TCN) for action recognition in time-series data. We also propose a Feature Map Decoder as part of the interpretation analysis, which outputs a representation of model's hidden variables in the same domain as the input. Such analysis empowers us to expose model's chara…
We present an interpretable deep model for fine-grained visual recognition. At the core of our method lies the integration of region-based part discovery and attribution within a deep neural network. Our model is trained using image-level object labels, and provides an interpretation of its results via the segmentation of object parts and the identification of their contributions towards classification. To facilitate the learning of object parts without direct supervision, we explore a simple prior of the occurrence of object parts. We demonstrate that this prior, when combined with our region…
Information Bottleneck-Guided Heterogeneous Graph Learning for Interpretable Neurodevelopmental Disorder Diagnosis
Department of Language Science and Technology, The Hong Kong Polytechnic University（香港理工大学语言科学与技术系）；Laboratory of Digital Image and Intelligent Computation, Shanghai Maritime University（上海 Maritime 大学数字图像与智能计算实验室）
Interpretable Single-View 3D Gaussian Splatting using Unsupervised Hierarchical Disentangled Representation Learning
Shanghai Jiao Tong University（上海交通大学）；Ningbo Institute of Digital Twin, Eastern Institute of Technology（东方理工大学宁波数字孪生研究院）；Hong Kong Polytechnic University（香港理工大学）
Fine-grained visual recognition is to classify objects with visually similar appearances into subcategories, which has made great progress with the development of deep CNNs. However, handling subtle differences between different subcategories still remains a challenge. In this paper, we propose to solve this issue in one unified framework from two aspects, i.e., constructing feature-level interrelationships, and capturing part-level discriminative features. This framework, namely PArt-guided Relational Transformers (PART), is proposed to learn the discriminative part features with an automatic…
Language Guided Concept Bottleneck Models for Interpretable Continual Learning
School of Computer Science and Engineering, Tianjin University of Technology（天津理工大学计算机科学与工程学院）；School of Information Science and Technology, University of Science and Technology of China（中国科学技术大学信息科学技术学院）；Institute of Automation, University of Chinese Academy of Sciences（中国科学院自动化研究所）
Fine-grained visual classification (FGVC) is becoming an important research field, due to its wide applications and the rapid development of computer vision technologies. The current state-of-the-art (SOTA) methods in the FGVC usually employ attention mechanisms to first capture the semantic parts and then discover their subtle differences between distinct classes. The channel-spatial attention mechanisms, which focus on the discriminative channels and regions simultaneously, have significantly improved the classification performance. However, the existing attention modules are poorly guided s…
Transitivity Recovering Decompositions: Interpretable and Robust Fine-Grained Relationships
University of Exeter（埃克塞特大学）；University of Trento（特伦托大学）；University of Tübingen（蒂宾根大学）；MPI for Informatics（马克斯·普朗克信息学研究所）；The Alan Turing Institute（艾伦·图灵研究所）；University of Surrey（萨里大学）
ConceptHash: Interpretable Fine-Grained Hashing via Concept Discovery
University of Surrey（萨里大学）；iFlyTek-Surrey Joint Research Centre on Artificial Intelligence（讯飞-萨里人工智能联合研究中心）；Surrey Institute for People-Centred Artificial Intelligence（萨里以人为本人工智能研究所）；CVSSP（视觉、语音和信号处理中心）
Eval3D: Interpretable and Fine-grained Evaluation for 3D Generation
Massachusetts Institute of Technology（麻省理工学院）；University of Washington（华盛顿大学）；New York University（纽约大学）；Wayve（Wayve公司）；Allen Institute for AI（艾伦人工智能研究所）；Cornell University（康奈尔大学）
Bosch Center for Artificial Intelligence (BCAI)（博世人工智能中心）；University of Bonn（波恩大学）
To verify and validate networks, it is essential to gain insight into their decisions, limitations as well as possible shortcomings of training data. In this work, we propose a post-hoc, optimization based visual explanation method, which highlights the evidence in the input image for a specific prediction. Our approach is based on a novel technique to defend against adversarial evidence (i.e. faulty evidence due to artefacts) by filtering gradients during optimization. The defense does not depend on human-tuned parameters. It enables explanations which are both fine-grained and preserve the c…
3D Gaussian Splatting for Fine-grained Urban Scene Reconstruction via Semantic-guided Density Control and Loss Design
School of Computer Science, China University of Geosciences (Wuhan)（中国地质大学（武汉）计算机学院）；School of Architecture and Urban Planning, Shenzhen University（深圳大学建筑与城市规划学院）；Guangdong Key Laboratory of Urban Informatics, Shenzhen University（深圳大学广东省城市信息学重点实验室）；MNR Key Laboratory for Geo-Environmental Monitoring of Great Bay Area, Shenzhen University（深圳大学自然资源部大湾区地理环境监测重点实验室）
Fudan University（复旦大学）；East China Normal University（华东师范大学）；Videt Lab（Videt实验室）；Zhongshan Hospital, Fudan University（复旦大学附属中山医院）
Document layout analysis (DLA) plays an important role in information extraction and document understanding. At present, document layout analysis has reached a milestone achievement, however, document layout analysis of non-Manhattan is still a challenge. In this paper, we propose an image layer modeling method to tackle this challenge. To measure the proposed image layer modeling method, we propose a manually-labeled non-Manhattan layout fine-grained segmentation dataset named FPD. As far as we know, FPD is the first manually-labeled non-Manhattan layout fine-grained segmentation dataset. To …
Principle-Guided Supervision for Interpretable Uncertainty in Medical Image Segmentation
School of Data Science, Fudan University（复旦大学数据科学学院）；Institute of Science and Technology for Brain-Inspired Intelligence, Fudan University（复旦大学脑启发智能科学研究院）；National Heart and Lung Institute, Imperial College London（伦敦帝国理工学院国家心脏和肺研究所）
We present a method for fine-grained face manipulation. Given a face image with an arbitrary expression, our method can synthesize another arbitrary expression by the same person. This is achieved by first fitting a 3D face model and then disentangling the face into a texture and a shape. We then learn different networks in these two spaces. In the texture space, we use a conditional generative network to change the appearance, and carefully design input formats and loss functions to achieve the best results. In the shape space, we use a fully connected network to predict the accurate shapes a…
While the fine-grained visual categorization (FGVC) problems have been greatly developed in the past years, the Ultra-fine-grained visual categorization (Ultra-FGVC) problems have been understudied. FGVC aims at classifying objects from the same species (very similar categories), while the Ultra-FGVC targets at more challenging problems of classifying images at an ultra-fine granularity where even human experts may fail to identify the visual difference. The challenges for Ultra-FGVC mainly comes from two aspects: one is that the Ultra-FGVC often arises overfitting problems due to the lack of …
Discriminative localization is essential for fine-grained image classification task, which devotes to recognizing hundreds of subcategories in the same basic-level category. Reflecting on discriminative regions of objects, key differences among different subcategories are subtle and local. Existing methods generally adopt a two-stage learning framework: The first stage is to localize the discriminative regions of objects, and the second is to encode the discriminative features for training classifiers. However, these methods generally have two limitations: (1) Separation of the two-stage learn…
VLMs Guided Interpretable Decision Making for Autonomous Driving
Department of Computer Science, Tulane University（路易斯安那大学计算机科学系）；Qualcomm（高通公司）；Department of Industrial and Systems Engineering, North Carolina State University（北卡罗来纳州立大学工业与系统工程系）
Fine-grained Text and Image Guided Point Cloud Completion with CLIP Model
Dalian Maritime University（大连海事大学）；Zhejiang Sci-Tech University（浙江理工大学）；Beijing University of Technology（北京工业大学）；Dalian University of Technology（大连理工大学）
Visually-Guided Controllable Medical Image Generation via Fine-Grained Semantic Disentanglement
Computer Science and Engineering, Northeastern University, Shenyang, China（东北大学计算机科学与工程系，中国沈阳）；Key Laboratory of Intelligent Computing in Medical Image of Ministry of Education, Northeastern University, Shenyang, China（教育部医学图像智能计算重点实验室，东北大学，中国沈阳）；National Frontiers Science Center for Industrial Intelligence and Systems Optimization, Shenyang, China（工业智能与系统优化国家级前沿科学中心，中国沈阳）；AiShiWeiLai AI Research, China（艾世维来人工智能研究，中国）；Amii, University of Alberta, Edmonton, Alberta, Canada（阿尔伯塔大学艾米人工智能研究所，加拿大埃德蒙顿，阿尔伯塔）
FGSGT: Saliency-Guided Siamese Network Tracker Based on Key Fine-Grained Feature Information for Thermal Infrared Target Tracking
College of Computer and Information Technology, China Three Gorges University（三峡大学计算机与信息学院）；Hubei Province Engineering Technology Research Center for Construction Quality Testing Equipment, China Three Gorges University（三峡大学湖北省建筑质量检测装备工程技术研究中心）；Nantong Institute of Technology（南通理工学院）；Wuhan College（武汉学院）
Expert Knowledge-Guided Decision Calibration for Accurate Fine-Grained Tree Species Classification
State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University（武汉大学测绘遥感信息工程国家重点实验室）；Hubei Luojia Laboratory（湖北珞珈实验室）
Scene Graph-guided SegCaptioning Transformer with Fine-grained Alignment for Controllable Video Segmentation and Captioning
College of Computer Science and Electronic Engineering, Hunan University（湖南大学计算机科学与电子工程学院）；School of Robotics and the National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University（机器人学院和机器人视觉感知与控制技术国家工程研究中心，湖南大学）；School of Computing and Artificial Intelligence, Southwest Jiaotong University（计算机与人工智能学院，西南交通大学）；CRRC Zhuzhou Institute Company Ltd.（中车株洲院有限公司）；Nanyang Technological University（南洋理工大学）
Fine-Grained Behavior and Lane Constraints Guided Trajectory Prediction Method
School of Mechanical Engineering, Zhejiang University（浙江大学机械工程学院）；Guangdong Provincial Key Laboratory of Fully Actuated System Control Theory and Technology, School of Automation and Intelligent Manufacturing, Southern University of Science and Technology（南方科技大学自动化与智能制造学院广东省全驱系统控制理论与技术重点实验室）；Leapmotor Technology（零跑科技）
