We propose a method for inferring human attributes (such as gender, hair style, clothes style, expression, action) from images of people under large variation of viewpoint, pose, appearance, articulation and occlusion. Convolutional Neural Nets (CNN) have been shown to perform very well on large scale object recognition problems. In the context of attribute classification, however, the signal is often subtle and it may cover only a small part of the image, while the image is dominated by the effects of pose and viewpoint. Discounting for pose variation would require training on very large labeled datasets which are not presently available. Part-based models, such as poselets and DPM have been shown to perform well for this problem but they are limited by shallow low-level features. We propose a new method which combines part-based models and deep learning by training pose-normalized CNNs. We show substantial improvement vs. state-of-the-art methods on challenging attribute classification tasks in unconstrained settings. Experiments confirm that our method outperforms both the best part-based methods on this problem and conventional CNNs trained on the full bounding box of the person.
Anomaly detection methods strive to discover patterns that differ from the norm in a semantic way. This goal is ambiguous as a data point differing from the norm by an attribute e.g., age, race or gender, may be considered anomalous by some operators while others may consider this attribute irrelevant. Breaking from previous research, we present a new anomaly detection method that allows operators to exclude an attribute from being considered as relevant for anomaly detection. Our approach then learns representations which do not contain information over the nuisance attributes. Anomaly scorin…
Interpretable Zero-Shot Learning with Locally-Aligned Vision-Language Model
Mohamed bin Zayed University of AI（莫扎伊德大学人工智能学院）；Huazhong University of Science and Technology（华中科技大学）；Australian National University（澳大利亚国立大学）；Linköping University（林雪平大学）
Learning Adaptive and Visually Aligned Conditions for Generative Zero-Shot Learning
School of Computer Science and Engineering, Southeast University, Nanjing, China（东南大学计算机科学与工程学院）；Qilu University of Technology (Shandong Academy of Sciences), Jinan, China（齐鲁工业大学（山东省科学院））
Facial attribute analysis in the real world scenario is very challenging mainly because of complex face variations. Existing works of analyzing face attributes are mostly based on the cropped and aligned face images. However, this result in the capability of attribute prediction heavily relies on the preprocessing of face detector. To address this problem, we present a novel jointly learned deep architecture for both facial attribute analysis and face detection. Our framework can process the natural images in the wild and our experiments on CelebA and LFWA datasets clearly show that the state-…
Institute of Computing Technology（计算技术研究所）；Peking University（北京大学）；University of Chinese Academy of Sciences（中国科学院大学）
Person re-identification is a challenging task due to various complex factors. Recent studies have attempted to integrate human parsing results or externally defined attributes to help capture human parts or important object regions. On the other hand, there still exist many useful contextual cues that do not fall into the scope of predefined human parts or attributes. In this paper, we address the missed contextual cues by exploiting both the accurate human parts and the coarse non-human parts. In our implementation, we apply a human parsing model to extract the binary human part masks \emph{…
University of Electronic Science and Technology of China（电子科技大学）；Southern University of Science and Technology（南方科技大学）
Zero-shot learning (ZSL) aims to recognize unseen classes based on the knowledge of seen classes. Previous methods focused on learning direct embeddings from global features to the semantic space in hope of knowledge transfer from seen classes to unseen classes. However, an unseen class shares local visual features with a set of seen classes and leveraging global visual features makes the knowledge transfer ineffective. To tackle this problem, we propose a Region Semantically Aligned Network (RSAN), which maps local features of unseen classes to their semantic attributes. Instead of using glob…
The goal of this paper is to retrieve an image based on instance, attribute and category similarity notions. Different from existing works, which usually address only one of these entities in isolation, we introduce a cooperative embedding to integrate them while preserving their specific level of semantic representation. An algebraic structure defines a superspace filled with instances. Attributes are axis-aligned to form subspaces, while categories influence the arrangement of similar instances. These relationships enable them to cooperate for their mutual benefits for image retrieval. We de…
NVIDIA（英伟达）；University of Toronto（多伦多大学）；Vector Institute（向量研究所）；MIT（麻省理工学院）
Modern image generative models show remarkable sample quality when trained on a single domain or class of objects. In this work, we introduce a generative adversarial network that can simultaneously generate aligned image samples from multiple related domains. We leverage the fact that a variety of object classes share common attributes, with certain geometric differences. We propose Polymorphic-GAN which learns shared features across all domains and a per-domain morph layer to morph shared features according to each domain. In contrast to previous works, our framework allows simultaneous mode…
School of Computer Science and Technology（计算机科学与技术学院）；East China Normal University（东华师范大学）；School of Design（设计学院）
Artistic image assessment has become a prominent research area in computer vision. In recent years, the field has witnessed a proliferation of datasets and methods designed to evaluate the aesthetic quality of paintings. However, most existing approaches focus solely on static final images, overlooking the dynamic and multi-stage nature of the artistic painting process. To address this gap, we propose a novel framework for human-aligned assessment of painting processes. Specifically, we introduce the Painting Process Assessment Dataset (PPAD), the first large-scale dataset comprising real and …
Vision and Security Technology (VAST) Lab（视觉与安全技术实验室）；University of Colorado, Colorado Springs（科罗拉多大学科罗拉多斯普林斯分校）
Facial attributes are soft-biometrics that allow limiting the search space, e.g., by rejecting identities with non-matching facial characteristics such as nose sizes or eyebrow shapes. In this paper, we investigate how the latest versions of deep convolutional neural networks, ResNets, perform on the facial attribute classification task. We test two loss functions: the sigmoid cross-entropy loss and the Euclidean loss, and find that for classification performance there is little difference between these two. Using an ensemble of three ResNets, we obtain the new state-of-the-art facial attribut…
Australian National University（澳大利亚国立大学）；Data61/CSIRO（Data61/联邦科学与工业研究组织）
Recovering a photorealistic face from an artistic portrait is a challenging task since crucial facial details are often distorted or completely lost in artistic compositions. To handle this loss, we propose an Attribute-guided Face Recovery from Portraits (AFRP) that utilizes a Face Recovery Network (FRN) and a Discriminative Network (DN). FRN consists of an autoencoder with residual block-embedded skip-connections and incorporates facial attribute vectors into the feature maps of input portraits at the bottleneck of the autoencoder. DN has multiple convolutional and fully-connected layers, an…
We present a novel attribute learning framework named Hypergraph-based Attribute Predictor (HAP). In HAP, a hypergraph is leveraged to depict the attribute relations in the data. Then the attribute prediction problem is casted as a regularized hypergraph cut problem in which HAP jointly learns a collection of attribute projections from the feature space to a hypergraph embedding space aligned with the attribute space. The learned projections directly act as attribute classifiers (linear and kernelized). This formulation leads to a very efficient approach. By considering our model as a multi-gr…
Style-Aligned Image Composition for Robust Detection of Abnormal Cells in Cytopathology
Zhejiang University（浙江大学）；University of Science and Technology of China（中国科学技术大学）；Suzhou Institute for Advance Research, USTC（中国科学技术大学苏州市先进研究院）；The Institute for Biomedical Engineering & Nano Science, Tongji University School of Medicine（同济大学医学院生物医学工程与纳米科学研究院）；Zhihui Medical Technology (Shanghai) Co., Ltd.（智辉医疗技术（上海）有限公司）
MGIMM: Multi-Granularity Instruction Multimodal Model for Attribute-Guided Remote Sensing Image Detailed Description
National Engineering Research Center for Multimedia Software（国家多媒体软件工程技术研究中心）；School of Computer Science, Wuhan University（武汉大学计算机学院）；Hubei Luojia Laboratory（湖北珞珈实验室）
The College of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics（南京航空航天大学人工智能学院）；The Key Laboratory of Brain-Machine Intelligence Technology, Ministry of Education（教育部脑机智能技术重点实验室）
University of Bristol（布里斯托大学）；University of Cambridge（剑桥大学）；Queen Mary University of London（伦敦玛丽女王大学）
In this paper, we present a novel single shot face-related task analysis method, called Face-SSD, for detecting faces and for performing various face-related (classification/regression) tasks including smile recognition, face attribute prediction and valence-arousal estimation in the wild. Face-SSD uses a Fully Convolutional Neural Network (FCNN) to detect multiple faces of different sizes and recognise/regress one or more face-related classes. Face-SSD has two parallel branches that share the same low-level filters, one branch dealing with face detection and the other one with face analysis t…
Controllable Image Captioning with Prompt-Conditioned Scene Rewards
POSTECH（浦项科技大学）；Graduate School of Artificial Intelligence, POSTECH（浦项科技大学人工智能研究生院）；Department of Computer Science and Engineering, POSTECH（浦项科技大学计算机科学与工程系）
Causal-Adapter: Taming Text-to-Image Diffusion for Faithful Counterfactual Generation
Centre for AI, DS\&AI, Astrazeneca, UK（英国阿斯利康人工智能中心）；Institute for Imaging, Data and Communications (IDCOM), School of Engineering, University of Edinburgh, Edinburgh, UK（爱丁堡大学工程学院影像、数据与通信研究所）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
FoolSDEdit: Deceptively Steering Your Edits Towards Targeted Attribute-aware Distribution
Zhejiang University（浙江大学）；Nanyang Technological University（南洋理工大学）；Institute of High Performance Computing (IHPC)（高性能计算研究所）；Centre for Frontier AI Research (CFAR)（前沿人工智能研究中心）
Object Transfiguration replaces an object in an image with another object from a second image. For example it can perform tasks like "putting exactly those eyeglasses from image A on the nose of the person in image B". Usage of exemplar images allows more precise specification of desired modifications and improves the diversity of conditional image generation. However, previous methods that rely on feature space operations, require paired data and/or appearance models for training or disentangling objects from background. In this work, we propose a model that can learn object transfiguration f…
