Learning Attentive Pairwise Interaction for Fine-Grained Classification
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Shenzhen Institute of Artificial Intelligence and Robotics for Society（深圳人工智能与机器人研究院）
Fine-grained classification is a challenging problem, due to subtle differences among highly-confused categories. Most approaches address this difficulty by learning discriminative representation of individual input image. On the other hand, humans can effectively identify contrastive clues by comparing image pairs. Inspired by this fact, this paper proposes a simple but effective Attentive Pairwise Interaction Network (API-Net), which can progressively recognize a pair of fine-grained images by interaction. Specifically, API-Net first learns a mutual feature vector to capture semantic differences in the input pair. It then compares this mutual vector with individual vectors to generate gates for each input image. These distinct gate vectors inherit mutual context on semantic differences, which allow API-Net to attentively capture contrastive clues by pairwise interaction between two images. Additionally, we train API-Net in an end-to-end manner with a score ranking regularization, which can further generalize API-Net by taking feature priorities into account. We conduct extensive experiments on five popular benchmarks in fine-grained classification. API-Net outperforms the recent SOTA methods, i.e., CUB-200-2011 (90.0%), Aircraft(93.9%), Stanford Cars (95.3%), Stanford Dogs (90.3%), and NABirds (88.1%).
University of Science and Technology of China（中国科学技术大学）；Microsoft Research Asia（微软亚洲研究院）
Video-based person re-identification (reID) aims at matching the same person across video clips. It is a challenging task due to the existence of redundancy among frames, newly revealed appearance, occlusion, and motion blurs. In this paper, we propose an attentive feature aggregation module, namely Multi-Granularity Reference-aided Attentive Feature Aggregation (MG-RAFA), to delicately aggregate spatio-temporal features into a discriminative video-level feature representation. In order to determine the contribution/importance of a spatial-temporal feature node, we propose to learn the attenti…
The recognition ability of human beings is developed in a progressive way. Usually, children learn to discriminate various objects from coarse to fine-grained with limited supervision. Inspired by this learning process, we propose a simple yet effective model for the Few-Shot Fine-Grained (FSFG) recognition, which tries to tackle the challenging fine-grained recognition task using meta-learning. The proposed method, named Pairwise Alignment Bilinear Network (PABN), is an end-to-end deep neural network. Unlike traditional deep bilinear networks for fine-grained classification, which adopt the s…
The Chinese University of Hong Kong（香港中文大学）；SenseTime Group Limited（商汤集团有限公司）
Pedestrian analysis plays a vital role in intelligent video surveillance and is a key component for security-centric computer vision systems. Despite that the convolutional neural networks are remarkable in learning discriminative features from images, the learning of comprehensive features of pedestrians for fine-grained tasks remains an open problem. In this study, we propose a new attention-based deep neural network, named as HydraPlus-Net (HP-net), that multi-directionally feeds the multi-level attention maps to different feature layers. The attentive deep features learned from the propose…
Faculty of Engineering and Information Technology, University of Technology Sydney（悉尼科技大学工程与信息技术学院）；School of Computer Science, The University of Adelaide（阿德莱德大学计算机学院）
Deep neural networks have demonstrated advanced abilities on various visual classification tasks, which heavily rely on the large-scale training samples with annotated ground-truth. However, it is unrealistic always to require such annotation in real-world applications. Recently, Few-Shot learning (FS), as an attempt to address the shortage of training samples, has made significant progress in generic classification tasks. Nonetheless, it is still challenging for current FS models to distinguish the subtle differences between fine-grained categories given limited training data. To filling the …
Beihang University（北京航空航天大学）；University of Adelaide（阿德莱德大学）；Griffith University（格里菲斯大学）；National Institute of Informatics（国立情报学研究所）；University of Tokyo（东京大学）
Person Re-IDentification (ReID) aims at re-identifying persons from different viewpoints across multiple cameras. Capturing the fine-grained appearance differences is often the key to accurate person ReID, because many identities can be differentiated only when looking into these fine-grained differences. However, most state-of-the-art person ReID approaches, typically driven by a triplet loss, fail to effectively learn the fine-grained features as they are focused more on differentiating large appearance differences. To address this issue, we introduce a novel pairwise loss function that enab…
The University of Queensland（昆士兰大学）；The University of New South Wales（新南威尔士大学）
Fine-grained visual recognition typically depends on modeling subtle difference from object parts. However, these parts often exhibit dramatic visual variations such as occlusions, viewpoints, and spatial transformations, making it hard to detect. In this paper, we present a novel attention-based model to automatically, selectively and accurately focus on critical object regions with higher importance against appearance variations. Given an image, two different Convolutional Neural Networks (CNNs) are constructed, where the outputs of two CNNs are correlated through bilinear pooling to simulta…
Recognizing fine-grained categories remains a challenging task, due to the subtle distinctions among different subordinate categories, which results in the need of abundant annotated samples. To alleviate the data-hungry problem, we consider the problem of learning novel categories from web data with the support of a clean set of base categories, which is referred to as weak-shot learning. In this setting, we propose a method called SimTrans to transfer pairwise semantic similarity from base categories to novel categories. Specifically, we firstly train a similarity net on clean data, and then…
Extremely Fine-Grained Visual Classification over Resembling Glyphs in the Wild
School of Computer and Information Engineering, Henan University（河南大学计算机与信息工程学院）；UPV/EHU（巴斯克大学）；IKERBASQUE, Basque Foundation for Science（巴斯克科学基金会（IKERBASQUE））
Compressed sensing (CS) leverages the sparsity prior to provide the foundation for fast magnetic resonance imaging (fastMRI). However, iterative solvers for ill-posed problems hinder their adaption to time-critical applications. Moreover, such a prior can be neither rich to capture complicated anatomical structures nor applicable to meet the demand of high-fidelity reconstructions in modern MRI. Inspired by the state-of-the-art methods in image generation, we propose a novel attention-based deep learning framework to provide high-quality MRI reconstruction. We incorporate large-field contextua…
Nanjing University of Science and Technology（南京理工大学）
Few-shot fine-grained recognition (FS-FGR) aims to recognize novel fine-grained categories with the help of limited available samples. Undoubtedly, this task inherits the main challenges from both few-shot learning and fine-grained recognition. First, the lack of labeled samples makes the learned model easy to overfit. Second, it also suffers from high intra-class variance and low inter-class differences in the datasets. To address this challenging task, we propose a two-stage background suppression and foreground alignment framework, which is composed of a background activation suppression (B…
University of Massachusetts, Amherst（马萨诸塞大学阿默斯特分校）
We present a framework for learning to describe fine-grained visual differences between instances using attribute phrases. Attribute phrases capture distinguishing aspects of an object (e.g., "propeller on the nose" or "door near the wing" for airplanes) in a compositional manner. Instances within a category can be described by a set of these phrases and collectively they span the space of semantic attributes for a category. We collect a large dataset of such phrases by asking annotators to describe several visual differences between a pair of instances within a category. We then learn to desc…
Semantic segmentation of remotely sensed images plays a crucial role in precision agriculture, environmental protection, and economic assessment. In recent years, substantial fine-resolution remote sensing images are available for semantic segmentation. However, due to the complicated information caused by the increased spatial resolution, state-of-the-art deep learning algorithms normally utilize complex network architectures for segmentation, which usually incurs high computational complexity. Specifically, the high-caliber performance of the convolutional neural network (CNN) heavily relies…
MAProtoNet: A Multi-scale Attentive Interpretable Prototypical Part Network for 3D Magnetic Resonance Imaging Brain Tumor Classification
Tokyo University of Agriculture and Technology (TUAT)（东京农工大学）；RIKEN Center for Advanced Intelligence Project (RIKEN AIP)（理化学研究所先进智能研究中心）；Graduate School of Medicine, Juntendo University（顺天堂大学医学院）；Faculty of Health Data Science, Juntendo University（顺天堂大学健康数据科学学部）；RIKEN（理化学研究所）
Massachusetts Institute of Technology（麻省理工学院）；Brigham Young University（杨百翰大学）；Harvard University（哈佛大学）
Fine-Grained Visual Classification (FGVC) datasets contain small sample sizes, along with significant intra-class variation and inter-class similarity. While prior work has addressed intra-class variation using localization and segmentation techniques, inter-class similarity may also affect feature learning and reduce classification performance. In this work, we address this problem using a novel optimization procedure for the end-to-end neural network training on FGVC tasks. Our procedure, called Pairwise Confusion (PC) reduces overfitting by intentionally {introducing confusion} in the activ…
Location embedding based pairwise distance learning for fine-grained diagnosis of urinary stones
School of Software, Northwestern Polytechnical University（西北工业大学软件学院）；Yangtze River Delta Research Institute of NPU（西北工业大学长三角研究院）；CSIRO Data61（联邦科学与工业研究组织数据61分部）；La Trobe University（拉筹伯大学）；School of Engineering, Shantou University（汕头大学工学院）；College of Intelligence and Computing, Tianjin University（天津大学智能与计算学部）；School of Information, Xiamen University（厦门大学信息学院）；AIDD, Faculty of Applied Science, Macao Polytechnic University（澳门理工大学应用科学学院AIDD）；Shuang-Ho Hospital, Taipei Medical University（台北医学大学双和医院）；National Yang Ming Chiao Tung University（国立阳明交通大学）；School of Design, The Hong Kong Polytechnic University（香港理工大学设计学院）；School of Medicine, College of Medicine, Taipei Medical University（台北医学大学医学院）
Stanford University（斯坦福大学）；Race Optimal（Race Optimal公司）；Ecole Polytechnique Federale de Lausanne (EPFL)（洛桑联邦理工学院（EPFL））
We present an interpretable framework for path prediction that leverages dependencies between agents' behaviors and their spatial navigation environment. We exploit two sources of information: the past motion trajectory of the agent of interest and a wide top-view image of the navigation scene. We propose a Clairvoyant Attentive Recurrent Network (CAR-Net) that learns where to look in a large image of the scene when solving the path prediction task. Our method can attend to any area, or combination of areas, within the raw image (e.g., road intersections) when predicting the trajectory of the …
Attributes Grouping and Mining Hashing for Fine-Grained Image Retrieval
Southeast University（东南大学）；Key Laboratory of Measurement and Control of Complex Systems of Engineering, Ministry of Education（教育部复杂工程系统测量与控制重点实验室）；Nanjing Enbo Technology Co., Ltd.（南京恩博科技有限公司）
HwaMei Hospital, University of Chinese Academy of Sciences（中国科学院大学华美医院）；Ningbo Institute of Life and Health Industry, UCAS（中国科学院大学宁波生命与健康产业研究院）；University of Hong Kong（香港大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Harbin Institute of Technology（哈尔滨工业大学）
In recent years, several works have adopted the convolutional neural network (CNN) to diagnose the avascular necrosis of the femoral head (AVNFH) based on X-ray images or magnetic resonance imaging (MRI). However, due to the tissue overlap, X-ray images are difficult to provide fine-grained features for early diagnosis. MRI, on the other hand, has a long imaging time, is more expensive, making it impractical in mass screening. Computed tomography (CT) shows layer-wise tissues, is faster to image, and is less costly than MRI. However, to our knowledge, there is no work on CT-based automated dia…
University of Science and Technology of China（中国科学技术大学）；Microsoft Research（微软研究院）；University of Rochester（罗切斯特大学）
Bilinear feature transformation has shown the state-of-the-art performance in learning fine-grained image representations. However, the computational cost to learn pairwise interactions between deep feature channels is prohibitively expensive, which restricts this powerful transformation to be used in deep neural networks. In this paper, we propose a deep bilinear transformation (DBT) block, which can be deeply stacked in convolutional neural networks to learn fine-grained image representations. The DBT block can uniformly divide input channels into several semantic groups. As bilinear transfo…
Locating discriminative parts plays a key role in fine-grained visual classification due to the high similarities between different objects. Recent works based on convolutional neural networks utilize the feature maps taken from the last convolutional layer to mine discriminative regions. However, the last convolutional layer tends to focus on the whole object due to the large receptive field, which leads to a reduced ability to spot the differences. To address this issue, we propose a novel Granularity-Aware Convolutional Neural Network (GA-CNN) that progressively explores discriminative feat…
URoadNet: Dual Sparse Attentive U-Net for Multiscale Road Network Extraction
College of Automation & College of Artificial Intelligence, Nanjing University of Posts and Telecommunications（南京邮电大学自动化学院与人工智能学院）；School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）；Jarvis Research Center, Tencent YouTu Lab（腾讯优图实验室贾维斯研究中心）
Zhejiang Sci-Tech University（浙江理工大学）；Zhejiang University（浙江大学）；Chongqing University（重庆大学）；Beijing Institute of Technology（北京理工大学）；Royal Brompton Hospital（皇家布朗普顿医院）；Imperial College London（伦敦帝国理工学院）
High-resolution magnetic resonance images can provide fine-grained anatomical information, but acquiring such data requires a long scanning time. In this paper, a framework called the Fused Attentive Generative Adversarial Networks(FA-GAN) is proposed to generate the super-resolution MR image from low-resolution magnetic resonance images, which can reduce the scanning time effectively but with high resolution MR images. In the framework of the FA-GAN, the local fusion feature block, consisting of different three-pass networks by using different convolution kernels, is proposed to extract image…
In this paper, we address the extraction of the fine-grained attributes of an instance as a `multi-attribute classification' problem. To this end, we propose an end-to-end architecture by adopting the bi-linear Convolutional Neural Network with the pairwise ranking loss. This is the first time such architecture is applied for the fine-grained attributes classification problem. We compared the proposed method with a competitive deep Convolutional Neural Network baseline. Extensive experiments show that the proposed method attains/outperforms the performance of compared baseline with significant…
Southwest Jiaotong University（西南交通大学）；National University of Singapore（新加坡国立大学）
Fine-grained object classification is a challenging task due to the subtle inter-class difference and large intra-class variation. Recently, visual attention models have been applied to automatically localize the discriminative regions of an image for better capturing critical difference and demonstrated promising performance. However, without consideration of the diversity in attention process, most of existing attention models perform poorly in classifying fine-grained objects. In this paper, we propose a diversified visual attention network (DVAN) to address the problems of fine-grained obj…
We present Implicit Two Hands (Im2Hands), the first neural implicit representation of two interacting hands. Unlike existing methods on two-hand reconstruction that rely on a parametric hand model and/or low-resolution meshes, Im2Hands can produce fine-grained geometry of two hands with high hand-to-hand and hand-to-image coherency. To handle the shape complexity and interaction context between two hands, Im2Hands models the occupancy volume of two hands - conditioned on an RGB image and coarse 3D keypoints - by two novel attention-based modules responsible for (1) initial occupancy estimation…
VersusQ: Pairwise Margin Reasoning for Generalizable Video Quality Assessment
The Chinese University of Hong Kong（香港中文大学）；WeChat Vision, Tencent Inc.（腾讯公司视觉部门）；Beijing Normal University（北京师范大学）；Peking University（北京大学）；The University of Hong Kong（香港大学）
DeepFashion is a widely used clothing dataset with 50 categories and more than overall 200k images where each image is annotated with fine-grained attributes. This dataset is often used for clothes recognition and although it provides comprehensive annotations, the attributes distribution is unbalanced and repetitive specially for training fine-grained attribute recognition models. In this work, we tailored DeepFashion for fine-grained attribute recognition task by focusing on each category separately. After selecting categories with sufficient number of images for training, we remove very sca…
Arena as Offline Reward: Efficient Fine-Grained Preference Optimization for Diffusion Models
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of California, Berkeley（加州大学伯克利分校）；University of California, Santa Barbara（加州大学圣巴巴拉分校）
Accurate and Scalable Multimodal Pathology Retrieval via Attentive Vision-Language Alignment
Department of Computer Science and Engineering, The Hong Kong University of Science and Technology（香港科学与技术大学计算机科学与工程系）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；Department of Radiology, Union Hospital, Tongji Medical College, Huazhong University of Science and Technology（华中科技大学同济医学院附属同济医院放射科）；Department of Pathology, Sir Run Run Shaw Hospital, School of Medicine, Zhejiang University（浙江大学医学院附属邵氏医院病理科）；Department of Pathology, The Central Hospital of Wuhan, Tongji Medical College, Huazhong University of Science and Technology（华中科技大学同济医学院附属武汉中心医院病理科）；Department of Pathology, The First Affiliated Hospital, School of Medicine, Zhejiang University（浙江大学医学院附属第一医院病理科）；Department of Chemical and Biological Engineering, The Hong Kong University of Science and Technology（香港科学与技术大学化学与生物工程系）；Division of Life Science, The Hong Kong University of Science and Technology（香港科学与技术大学生命科学系）
Looking Back and Forth: Cross-Image Attention Calibration and Attentive Preference Learning for Multi-Image Hallucination Mitigation
Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Harbin Institute of Technology（哈尔滨工业大学）；Beijing Institute of Technology（北京理工大学）；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学深圳分校）
Norwegian University of Science and Technology（挪威科技大学）；University Federico II of Naples（那不勒斯费德里科二世大学）
A fast-paced development of DeepFake generation techniques challenge the detection schemes designed for known type DeepFakes. A reliable Deepfake detection approach must be agnostic to generation types, which can present diverse quality and appearance. Limited generalizability across different generation schemes will restrict the wide-scale deployment of detectors if they fail to handle unseen attacks in an open set scenario. We propose a new approach, Multi-Channel Xception Attention Pairwise Interaction (MCX-API), that exploits the power of pairwise learning and complementary information fro…
Huazhong University of Science and Technology（华中科技大学）；Microsoft Research AI（微软研究院AI）
In this paper, we investigate a novel problem of telling the difference between image pairs in natural language. Compared to previous approaches for single image captioning, it is challenging to fetch linguistic representation from two independent visual information. To this end, we have proposed an effective encoder-decoder caption framework based on Hyper Convolution Net. In addition, a series of novel feature fusing techniques for pairwise visual information fusing are introduced and a discriminating referee is proposed to evaluate the pipeline. Because of the lack of appropriate datasets t…
University of Science and Technology of China（中国科学技术大学）；Microsoft Research（微软研究院）
We present variational generative adversarial networks, a general learning framework that combines a variational auto-encoder with a generative adversarial network, for synthesizing images in fine-grained categories, such as faces of a specific person or objects in a category. Our approach models an image as a composition of label and latent attributes in a probabilistic model. By varying the fine-grained category label fed into the resulting generative model, we can generate images in a specific category with randomly drawn values on a latent attribute vector. Our approach has two novel aspec…
Tsinghua University（清华大学）；Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）
Most existing action quality assessment methods rely on the deep features of an entire video to predict the score, which is less reliable due to the non-transparent inference process and poor interpretability. We argue that understanding both high-level semantics and internal temporal structures of actions in competitive sports videos is the key to making predictions accurate and interpretable. Towards this goal, we construct a new fine-grained dataset, called FineDiving, developed on diverse diving events with detailed annotations on action procedures. We also propose a procedure-aware approa…
