Fine-grained visual classification is a challenging task that recognizes the sub-classes belonging to the same meta-class. Large inter-class similarity and intra-class variance is the main challenge of this task. Most exiting methods try to solve this problem by designing complex model structures to explore more minute and discriminative regions. In this paper, we argue that mining multi-regional multi-grained features is precisely the key to this task. Specifically, we introduce a new loss function, termed top-down spatial attention loss (TDSA-Loss), which contains a multi-stage channel constrained module and a top-down spatial attention module. The multi-stage channel constrained module aims to make the feature channels in different stages category-aligned. Meanwhile, the top-down spatial attention module uses the attention map generated by high-level aligned feature channels to make middle-level aligned feature channels to focus on particular regions. Finally, we can obtain multiple discriminative regions on high-level feature channels and obtain multiple more minute regions within these discriminative regions on middle-level feature channels. In summary, we obtain multi-regional multi-grained features. Experimental results over four widely used fine-grained image classification datasets demonstrate the effectiveness of the proposed method. Ablative studies further show the superiority of two modules in the proposed method. Codes are available at: https://github.com/dongliangchang/Top-Down-Spatial-Attention-Loss.
RAGMesh with FaME-G2E: Long-Form Text-Driven 3D Face Generation and Editing
Beihang University（北京航空航天大学）；Pengcheng Laboratory（鹏城实验室）；Shanxi University of Finance and Economics（山西财经大学）；North China University of Technology（北方工业大学）；Cardiff University（卡迪夫大学）；Beijing Institute of Technology（北京理工大学）
Accurate whole-body multi-person pose estimation and tracking is an important yet challenging topic in computer vision. To capture the subtle actions of humans for complex behavior analysis, whole-body pose estimation including the face, body, hand and foot is essential over conventional body-only pose estimation. In this paper, we present AlphaPose, a system that can perform accurate whole-body pose estimation and tracking jointly while running in realtime. To this end, we propose several new techniques: Symmetric Integral Keypoint Regression (SIKR) for fast and fine localization, Parametric …
Robust Saliency-Aware Distillation for Few-shot Fine-grained Visual Recognition
School of Computer Science and Engineering, South China University of Technology（华南理工大学计算机科学与工程学院）；Brain and Affective Cognitive Research Center, Pazhou Lab（琶洲实验室脑与情感认知研究中心）；Engineering Research Center of the Ministry of Education on Health Intelligent Perception and Paralleled Digital-Human（教育部健康智能感知与平行数字人工程研究中心）
Tencent AI Lab（腾讯人工智能实验室）；Tsinghua Shenzhen International Graduate School（清华大学深圳国际研究生院）；South China University of Technology（华南理工大学）
We present a novel paradigm for high-fidelity face swapping that faithfully preserves the desired subtle geometry and texture details. We rethink face swapping from the perspective of fine-grained face editing, \textit{i.e., ``editing for swapping'' (E4S)}, and propose a framework that is based on the explicit disentanglement of the shape and texture of facial components. Following the E4S principle, our framework enables both global and local swapping of facial features, as well as controlling the amount of partial swapping specified by the user. Furthermore, the E4S paradigm is inherently ca…
Extracting discriminative features plays a crucial role in the fine-grained visual classification task. Most of the existing methods focus on developing attention or augmentation mechanisms to achieve this goal. However, addressing the ambiguity in the top-k prediction classes is not fully investigated. In this paper, we introduce a Self Assessment Classifier, which simultaneously leverages the representation of the image and top-k prediction classes to reassess the classification results. Our method is inspired by continual learning with coarse-grained and fine-grained classifiers to increase…
E4S: Fine-grained Face Swapping via Editing With Regional GAN Inversion
Tsinghua Shenzhen International Graduate School（清华大学深圳国际研究生院）；South China University of Technology（华南理工大学）；The University of Hong Kong（香港大学）；Tencent AI Lab（腾讯AI实验室）
Every day, humans perform many closely related activities that involve subtle discriminative motions, such as putting on a shirt vs. putting on a jacket, or shaking hands vs. giving a high five. Activity recognition by ethical visual AI could provide insights into our patterns of daily life, however existing activity recognition datasets do not capture the massive diversity of these human activities around the world. To address this limitation, we introduce Collector, a free mobile app to record video while simultaneously annotating objects and activities of consented subjects. This new data c…
FOCUS: Fine-grained Optimization with Semantic Guided Understanding for Pedestrian Attributes Recognition
School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所基础模型研究中心）；Peng Cheng Laboratory（鹏城实验室）；Wuhan AI Research（武汉人工智能研究院）；Objecteye Inc.（Objecteye公司）
ByteDance AI Lab（字节跳动人工智能实验室）；Johns Hopkins University（约翰斯·霍普金斯大学）
Search space design is very critical to neural architecture search (NAS) algorithms. We propose a fine-grained search space comprised of atomic blocks, a minimal search unit that is much smaller than the ones used in recent NAS algorithms. This search space allows a mix of operations by composing different types of atomic blocks, while the search space in previous methods only allows homogeneous operations. Based on this search space, we propose a resource-aware architecture search framework which automatically assigns the computational resources (e.g., output channel numbers) for each operati…
Edge Hill University（边山大学）；Aberystwyth University（阿伯里斯特威斯大学）
Affect is often expressed via non-verbal body language such as actions/gestures, which are vital indicators for human behaviors. Recent studies on recognition of fine-grained actions/gestures in monocular images have mainly focused on modeling spatial configuration of body parts representing body pose, human-objects interactions and variations in local appearance. The results show that this is a brittle approach since it relies on accurate body parts/objects detection. In this work, we argue that there exist local discriminative semantic regions, whose "informativeness" can be evaluated by the…
EliGen: Entity-Level Controlled Image Generation with Regional Attention
College of Control Science and Engineering, Zhejiang University（浙江大学控制科学与工程学院）；ModelScope Team, Alibaba Group Inc.（阿里巴巴集团通义千问团队）；East China Normal University（华东师范大学）
Hong Kong University of Science and Technology（香港科技大学）；Sun Yat-sen University（中山大学）
Unsupervised large-scale vision-language pre-training has shown promising advances on various downstream tasks. Existing methods often model the cross-modal interaction either via the similarity of the global feature of each modality which misses sufficient information, or finer-grained interactions using cross/self-attention upon visual and textual tokens. However, cross/self-attention suffers from inferior efficiency in both training and inference. In this paper, we introduce a large-scale Fine-grained Interactive Language-Image Pre-training (FILIP) to achieve finer-level alignment through a…
EviViT: Evidence-Adaptive Vision Transformers for Fine-Grained Perception
Tsinghua University（清华大学）；Peng Cheng Laboratory（鹏城实验室）；The Hong Kong University of Science and Technology（香港科技大学）；LMMs-Lab；Beijing Institute of Technology（北京理工大学）；University of New South Wales（新南威尔士大学）
Optimizing Vision-Language Consistency via Cross-Layer Regional Attention Alignment
School of Medicine, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）医学院）；Shenyang Institute of Automation, Chinese Academy of Sciences（中国科学院沈阳自动化研究所）；Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Wave and Machine Intelligence Department, Technology Innovation Institute（技术创新研究院波浪与机器智能部门）；University of the Chinese Academy of Sciences（中国科学院大学）；McGill University（麦吉尔大学）
Cross-media retrieval is to return the results of various media types corresponding to the query of any media type. Existing researches generally focus on coarse-grained cross-media retrieval. When users submit an image of "Slaty-backed Gull" as a query, coarse-grained cross-media retrieval treats it as "Bird", so that users can only get the results of "Bird", which may include other bird species with similar appearance (image and video), descriptions (text) or sounds (audio), such as "Herring Gull". Such coarse-grained cross-media retrieval is not consistent with human lifestyle, where we gen…
Knexus Research Corporation（Knexus研究公司）；University of Nevada, Las Vegas（内华达大学拉斯维加斯分校）；Naval Research Laboratory（海军研究实验室）
Although recent advances in regional Convolutional Neural Networks (CNNs) enable them to outperform conventional techniques on standard object detection and classification tasks, their response time is still slow for real-time performance. To address this issue, we propose a method for region proposal as an alternative to selective search, which is used in current state-of-the art object detection algorithms. We evaluate our Keypoint Density-based Region Proposal (KDRP) approach and show that it speeds up detection and classification on fine-grained tasks by 100% versus the existing selective …
ReBA-Pred-Net: Weakly-Supervised Regional Brain Age Prediction on MRI
University of Science and Technology of China（科学技术大学）；China University of Petroleum (East China)（中国石油大学（华东））；Northwestern Polytechnical University（西北工业大学）；Beijing Jiaotong University（北京交通大学）；Beijing Tiantan Hospital（北京天坛医院）
Hikvision Research Institute（海康威视研究院）；Zhejiang University（浙江大学）
In facial action unit (AU) recognition tasks, regional feature learning and AU relation modeling are two effective aspects which are worth exploring. However, the limited representation capacity of regional features makes it difficult for relation models to embed AU relationship knowledge. In this paper, we propose a novel multi-level adaptive ROI and graph learning (MARGL) framework to tackle this problem. Specifically, an adaptive ROI learning module is designed to automatically adjust the location and size of the predefined AU regions. Meanwhile, besides relationship between AUs, there exis…
SurfAge-Net: A Hierarchical Surface-Based Network for Interpretable Fine-Grained Brain Age Prediction
Gansu Provincial Key Laboratory of Wearable Computing, School of Information Science and Engineering, Lanzhou University（甘肃省可穿戴计算重点实验室，信息科学与工程学院，兰州大学）；Department of Medical Imaging Center, Gansu Provincial Maternity and Child-Care Hospital（医学影像中心，甘肃省妇幼保健院）；Department of Magnetic Resonance, Lanzhou University Second Hospital（磁共振科，兰州大学第二医院）；Department of Computer Science and Technology, Harbin Institute of Technology (Shenzhen)（计算机科学与技术系，哈尔滨工业大学（深圳））；Key Laboratory for Biomedical Engineering of Ministry of Education, Department of Biomedical Engineering, College of Biomedical Engineering & Instrument Science, Zhejiang University（教育部生物医学工程重点实验室，生物医学工程学院，浙江大学）
Institute of Computer Science and Technology, Peking University（北京大学计算机科学技术研究所）
Fine-grained image classification is to recognize hundreds of subcategories in each basic-level category. Existing methods employ discriminative localization to find the key distinctions among subcategories. However, they generally have two limitations: (1) Discriminative localization relies on region proposal methods to hypothesize the locations of discriminative regions, which are time-consuming. (2) The training of discriminative localization depends on object or part annotations, which are heavily labor-consuming. It is highly challenging to address the two key limitations simultaneously, …
Discriminative localization is essential for fine-grained image classification task, which devotes to recognizing hundreds of subcategories in the same basic-level category. Reflecting on discriminative regions of objects, key differences among different subcategories are subtle and local. Existing methods generally adopt a two-stage learning framework: The first stage is to localize the discriminative regions of objects, and the second is to encode the discriminative features for training classifiers. However, these methods generally have two limitations: (1) Separation of the two-stage learn…
We present a simple deep learning framework to simultaneously predict keypoint locations and their respective visibilities and use those to achieve state-of-the-art performance for fine-grained classification. We show that by conditioning the predictions on object proposals with sufficient image support, our method can do well without complicated spatial reasoning. Instead, inference methods with robustness to outliers, yield state-of-the-art for keypoint localization. We demonstrate the effectiveness of our accurate keypoint localization and visibility prediction on the fine-grained bird reco…
In this paper, we tackle the problem of visual categorization of dog breeds, which is a surprisingly challenging task due to simultaneously present low interclass distances and high intra-class variances. Our approach combines several techniques well known in our community but often not utilized for fine-grained recognition: (1) automatic segmentation, (2) efficient part detection, and (3) combination of multiple features. In particular, we demonstrate that a simple head detector embedded in an off-the-shelf recognition pipeline can improve recognition accuracy quite significantly, highlightin…
CoMLP: Cooperatively-Gated MLPs for Fine-Grained Cross-Modal Information Fusion in Medical Image Segmentation
Institute of Translational Medicine, Shanghai Jiao Tong University（上海交通大学转化医学研究院）；Zhongguancun Academy（中关村学院）；Zhongguancun Institute of Artificial Intelligence（中关村人工智能研究院）；School of Computer Science, the University of Sydney（悉尼大学计算机学院）；China United Network Communications Group Co., Ltd.（中国联合网络通信集团有限公司）
In this paper, we propose a multi-stage and high-resolution model for image synthesis that uses fine-grained attributes and masks as input. With a fine-grained attribute, the proposed model can detailedly constrain the features of the generated image through rich and fine-grained semantic information in the attribute. With mask as prior, the model in this paper is constrained so that the generated images conform to visual senses, which will reduce the unexpected diversity of samples generated from the generative adversarial network. This paper also proposes a scheme to improve the discriminato…
Fine-grained image recognition is a challenging computer vision problem, due to the small inter-class variations caused by highly similar subordinate categories, and the large intra-class variations in poses, scales and rotations. In this paper, we propose a novel end-to-end Mask-CNN model without the fully connected layers for fine-grained recognition. Based on the part annotations of fine-grained images, the proposed model consists of a fully convolutional network to both locate the discriminative parts (e.g., head and torso), and more importantly generate object/part masks for selecting use…
CoF: Coarse to Fine-Grained Image Understanding for Multi-modal Large Language Models
School of Automation, Northwestern Polytechnical University（西北工业大学自动化学院）；School of Cybersecurity, Northwestern Polytechnical University（西北工业大学网络空间安全学院）；Alibaba Group（阿里巴巴集团）；Southeast University（东南大学）；Advanced Ocean Institute of Southeast University（东南大学先进海洋研究院）；Zhejiang University of Technology（浙江工业大学）；Binjiang Institute of Artificial Intelligence（滨江人工智能研究院）
Fine-grained image hashing is a challenging problem due to the difficulties of discriminative region localization and hash code generation. Most existing deep hashing approaches solve the two tasks independently. While these two tasks are correlated and can reinforce each other. In this paper, we propose a deep fine-grained hashing to simultaneously localize the discriminative regions and generate the efficient binary codes. The proposed approach consists of a region localization module and a hash coding module. The region localization module aims to provide informative regions to the hash cod…
University of Technology, Sydney（悉尼科技大学）；Shanghai Jiao Tong University（上海交通大学）
In the context of fine-grained visual categorization, the ability to interpret models as human-understandable visual manuals is sometimes as important as achieving high classification accuracy. In this paper, we propose a novel Part-Stacked CNN architecture that explicitly explains the fine-grained recognition process by modeling subtle differences from object parts. Based on manually-labeled strong part annotations, the proposed architecture consists of a fully convolutional network to locate multiple object parts and a two-stream classification network that en- codes object-level and part-le…
Australian Centre for Robotic Vision（澳大利亚机器人视觉中心）；Queensland University of Technology (QUT)（昆士兰科技大学）；University of Queensland（昆士兰大学）；NICTA（澳大利亚国家信息与通信技术研究院）
We present a novel deep convolutional neural network (DCNN) system for fine-grained image classification, called a mixture of DCNNs (MixDCNN). The fine-grained image classification problem is characterised by large intra-class variations and small inter-class variations. To overcome these problems our proposed MixDCNN system partitions images into K subsets of similar images and learns an expert DCNN for each subset. The output from each of the K DCNNs is combined to form a single classification decision. In contrast to previous techniques, we provide a formulation to perform joint end-to-end …
We introduce the task of spotting temporally precise, fine-grained events in video (detecting the precise moment in time events occur). Precise spotting requires models to reason globally about the full-time scale of actions and locally to identify subtle frame-to-frame appearance and motion differences that identify events during these actions. Surprisingly, we find that top performing solutions to prior video understanding tasks such as action detection and segmentation do not simultaneously meet both requirements. In response, we propose E2E-Spot, a compact, end-to-end model that performs w…
Fine-grained image recognition is very challenging due to the difficulty of capturing both semantic global features and discriminative local features. Meanwhile, these two features are not easy to be integrated, which are even conflicting when used simultaneously. In this paper, a retrieval-based coarse-to-fine framework is proposed, where we re-rank the TopN classification results by using the local region enhanced embedding features to improve the Top1 accuracy (based on the observation that the correct category usually resides in TopN results). To obtain the discriminative regions for disti…
Boosting Single-domain Generalized Object Detection via Vision-Language Knowledge Interaction
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences（先进交叉学科研究院，中国科学院大学）；Institute of Microelectronics of the Chinese Academy of Sciences（中国科学院微电子研究所）
