Attaching attributes (such as color, shape, state, action) to object categories is an important computer vision problem. Attribute prediction has seen exciting recent progress and is often formulated as a multi-label classification problem. Yet significant challenges remain in: 1) predicting diverse attributes over multiple categories, 2) modeling attributes-category dependency, 3) capturing both global and local scene context, and 4) predicting attributes of objects with low pixel-count. To address these issues, we propose a novel multi-category attribute prediction deep architecture named GlideNet, which contains three distinct feature extractors. A global feature extractor recognizes what objects are present in a scene, whereas a local one focuses on the area surrounding the object of interest. Meanwhile, an intrinsic feature extractor uses an extension of standard convolution dubbed Informed Convolution to retrieve features of objects with low pixel-count. GlideNet uses gating mechanisms with binary masks and its self-learned category embedding to combine the dense embeddings. Collectively, the Global-Local-Intrinsic blocks comprehend the scene's global context while attending to the characteristics of the local object of interest. Finally, using the combined features, an interpreter predicts the attributes, and the length of the output is determined by the category, thereby removing unnecessary attributes. GlideNet can achieve compelling results on two recent and challenging datasets -- VAW and CAR -- for large-scale attribute prediction. For instance, it obtains more than 5\% gain over state of the art in the mean recall (mR) metric. GlideNet's advantages are especially apparent when predicting attributes of objects with low pixel counts as well as attributes that demand global context understanding. Finally, we show that GlideNet excels in training starved real-world scenarios.
Self-driving vehicles are the future of transportation. With current advancements in this field, the world is getting closer to safe roads with almost zero probability of having accidents and eliminating human errors. However, there is still plenty of research and development necessary to reach a level of robustness. One important aspect is to understand a scene fully including all details. As some characteristics (attributes) of objects in a scene (drivers' behavior for instance) could be imperative for correct decision making. However, current algorithms suffer from low-quality datasets with…
Material attributes have been shown to provide a discriminative intermediate representation for recognizing materials, especially for the challenging task of recognition from local material appearance (i.e., regardless of object and scene context). In the past, however, material attributes have been recognized separately preceding category recognition. In contrast, neuroscience studies on material perception and computer vision research on object and place recognition have shown that attributes are produced as a by-product during the category recognition process. Does the same hold true for ma…
Understanding product attributes plays an important role in improving online shopping experience for customers and serves as an integral part for constructing a product knowledge graph. Most existing methods focus on attribute extraction from text description or utilize visual information from product images such as shape and color. Compared to the inputs considered in prior works, a product image in fact contains more information, represented by a rich mixture of words and visual clues with a layout carefully designed to impress customers. This work proposes a more inclusive framework that fu…
University of Chinese Academy of Sciences（中国科学院大学）；Peng Cheng Laboratory（鹏城实验室）；Huawei Cloud（华为云）
Few-shot image generation is a challenging task even using the state-of-the-art Generative Adversarial Networks (GANs). Due to the unstable GAN training process and the limited training data, the generated images are often of low quality and low diversity. In this work, we propose a new editing-based method, i.e., Attribute Group Editing (AGE), for few-shot image generation. The basic assumption is that any image is a collection of attributes and the editing direction for a specific attribute is shared across all categories. AGE examines the internal representation learned in GANs and identifi…
How to learn a discriminative fine-grained representation is a key point in many computer vision applications, such as person re-identification, fine-grained classification, fine-grained image retrieval, etc. Most of the previous methods focus on learning metrics or ensemble to derive better global representation, which are usually lack of local information. Based on the considerations above, we propose a novel Attribute-Aware Attention Model ($A^3M$), which can learn local attribute representation and global category representation simultaneously in an end-to-end manner. The proposed model co…
The goal of this paper is to retrieve an image based on instance, attribute and category similarity notions. Different from existing works, which usually address only one of these entities in isolation, we introduce a cooperative embedding to integrate them while preserving their specific level of semantic representation. An algebraic structure defines a superspace filled with instances. Attributes are axis-aligned to form subspaces, while categories influence the arrangement of similar instances. These relationships enable them to cooperate for their mutual benefits for image retrieval. We de…
Tencent Data Platform（腾讯数据平台）；University of Science and Technology of China（中国科学技术大学）
Generalized Zero-Shot Learning (GZSL) aims to recognize new categories with auxiliary semantic information,e.g., category attributes. In this paper, we handle the critical issue of domain shift problem, i.e., confusion between seen and unseen categories, by progressively improving cross-domain transferability and category discriminability of visual representations. Our approach, named Dual Progressive Prototype Network (DPPN), constructs two types of prototypes that record prototypical visual patterns for attributes and categories, respectively. With attribute prototypes, DPPN alternately sear…
University of Amsterdam（阿姆斯特丹大学）；Columbia University（哥伦比亚大学）
This paper aims for generic instance search from one example where the instance can be an arbitrary object like shoes, not just near-planar and one-sided instances like buildings and logos. First, we evaluate state-of-the-art instance search methods on this problem. We observe that what works for buildings loses its generality on shoes. Second, we propose to use automatically learned category-specific attributes to address the large appearance variations present in generic instance search. Searching among instances from the same category as the query, the category-specific attributes outperfor…
North University of China（中北大学）；Southwest Petroleum University（西南石油大学）；Shanxi Research Center of Software Engineering Technology（山西省软件工程技术研究中心）
Scene classification has established itself as a challenging research problem. Compared to images of individual objects, scene images could be much more semantically complex and abstract. Their difference mainly lies in the level of granularity of recognition. Yet, image recognition serves as a key pillar for the good performance of scene recognition as the knowledge attained from object images can be used for accurate recognition of scenes. The existing scene recognition methods only take the category label of the scene into consideration. However, we find that the contextual information that…
University of Chinese Academy of Sciences（中国科学院大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Peng Cheng Laboratory（鹏城实验室）；Huawei Cloud EI Innovation Lab（华为云EI创新实验室）
Few-shot image generation aims to generate data of an unseen category based on only a few samples. Apart from basic content generation, a bunch of downstream applications hopefully benefit from this task, such as low-data detection and few-shot classification. To achieve this goal, the generated images should guarantee category retention for classification beyond the visual quality and diversity. In our preliminary work, we present an ``editing-based'' framework Attribute Group Editing (AGE) for reliable few-shot image generation, which largely improves the generation performance. Nevertheless…
In principle, zero-shot learning makes it possible to train a recognition model simply by specifying the category's attributes. For example, with classifiers for generic attributes like \emph{striped} and \emph{four-legged}, one can construct a classifier for the zebra category by enumerating which properties it possesses---even without providing zebra training images. In practice, however, the standard zero-shot paradigm suffers because attribute predictions in novel images are hard to get right. We propose a novel random forest approach to train zero-shot models that explicitly accounts for …
We propose a method that learns a discriminative yet semantic space for object categorization, where we also embed auxiliary semantic entities such as supercategories and attributes. Contrary to prior work which only utilized them as side information, we explicitly embed the semantic entities into the same space where we embed categories, which enables us to represent a category as their linear combination. By exploiting such a unified model for semantics, we enforce each category to be represented by a supercategory + sparse combination of attributes, with an additional exclusive regularizati…
We explore object detection with two attributes: color and material. The task aims to simultaneously detect objects and infer their color and material. A straight-forward approach is to add attribute heads at the very end of a usual object detection pipeline. However, we observe that the two goals are in conflict: Object detection should be attribute-independent and attributes be largely object-independent. Features computed by a standard detection network entangle the category and attribute features; we disentangle them by the use of a two-stream model where the category and attribute feature…
Tsinghua University（清华大学）；University of Iowa（爱荷华大学）
Attributes possess appealing properties and benefit many computer vision problems, such as object recognition, learning with humans in the loop, and image retrieval. Whereas the existing work mainly pursues utilizing attributes for various computer vision problems, we contend that the most basic problem---how to accurately and robustly detect attributes from images---has been left under explored. Especially, the existing work rarely explicitly tackles the need that attribute detectors should generalize well across different categories, including those previously unseen. Noting that this is ana…
University of Chinese Academy of Sciences（中国科学院大学）；Institute of Computing Technology, CAS（中国科学院计算技术研究所）
Recent years have seen more and more demand for a unified framework to address multiple realistic image retrieval tasks concerning both category and attributes. Considering the scale of modern datasets, hashing is favorable for its low complexity. However, most existing hashing methods are designed to preserve one single kind of similarity, thus improper for dealing with the different tasks simultaneously. To overcome this limitation, we propose a new hashing method, named Dual Purpose Hashing (DPH), which jointly preserves the category and attribute similarities by exploiting the Convolutiona…
School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）；Department of Engineering Science, University of Oxford（牛津大学工程科学系）；School of Engineering, Newcastle University（纽卡斯尔大学工程学院）
The recent advance in deep generative models outlines a promising perspective in the realm of Zero-Shot Learning (ZSL). Most generative ZSL methods use category semantic attributes plus a Gaussian noise to generate visual features. After generating unseen samples, this family of approaches effectively transforms the ZSL problem into a supervised classification scheme. However, the existing models use a single semantic attribute, which contains the complete attribute information of the category. The generated data also carry the complete attribute information, but in reality, visual samples usu…
Johns Hopkins University（约翰斯·霍普金斯大学）；Univ. of Maryland, College Park（马里兰大学帕克分校）；NIST（美国国家标准与技术研究院）
Face recognition networks generally demonstrate bias with respect to sensitive attributes like gender, skintone etc. For gender and skintone, we observe that the regions of the face that a network attends to vary by the category of an attribute. This might contribute to bias. Building on this intuition, we propose a novel distillation-based approach called Distill and De-bias (D&D) to enforce a network to attend to similar face regions, irrespective of the attribute category. In D&D, we train a teacher network on images from one category of an attribute; e.g. light skintone. Then distilling in…
Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）
Predicting the target of visual search from eye fixation (gaze) data is a challenging problem with many applications in human-computer interaction. In contrast to previous work that has focused on individual instances as a search target, we propose the first approach to predict categories and attributes of search targets based on gaze data. However, state of the art models for categorical recognition, in general, require large amounts of training data, which is prohibitive for gaze data. To address this challenge, we propose a novel Gaze Pooling Layer that integrates gaze information into CNN-…
TAGE: Trustworthy Attribute Group Editing for Stable Few-shot Image Generation
Guangdong University of Technology（广东工业大学）；Macao Polytechnic University（澳门理工大学）；Huizhou University（惠州学院）；Guangdong University of Foreign Studies（广东外语外贸大学）
In this paper, we consider the problem of simultaneously detecting objects and inferring their visual attributes in an image, even for those with no manual annotations provided at the training stage, resembling an open-vocabulary scenario. To achieve this goal, we make the following contributions: (i) we start with a naive two-stage approach for open-vocabulary object detection and attribute classification, termed CLIP-Attr. The candidate objects are first proposed with an offline RPN and later classified for semantic category and attributes; (ii) we combine all available datasets and train wi…
DSAA: Dual-Stage Attribute Activation for Fine-grained Open Vocabulary Detection
Beijing University of Posts and Telecommunications（北京邮电大学）；Beijing E-Hualu Information Technology Co., Ltd.（北京亿华鲁信息技术有限公司）；State Key Laboratory of General Artificial Intelligence, BIGAI, Beijing, China（通用人工智能国家重点实验室，BIGAI，北京，中国）
CREST: Cross-modal Resonance through Evidential Deep Learning for Enhanced Zero-Shot Learning
TeleAI；Northwestern Polytechnical University（西北工业大学）；The University of Hong Kong（香港大学）；University of Science and Technology of China（中国科学技术大学）；Zhejiang University（浙江大学）
SemPT: Semantic Prompt Tuning for Vision-Language Models
School of Computer Science and Artificial Intelligence, Wuhan Textile University（武汉纺织大学计算机科学与人工智能学院）；School of Remote Sensing and Information Engineering, Wuhan University（武汉大学遥感与信息工程学院）
School of Traffic and Transportation, Beijing Jiaotong University（交通与运输学院，北京交通大学）；Foundation Modal Research Center, Institute of Automation, Chinese Academy of Sciences（基础模态研究中心，自动化研究所）；Queen Mary School Hainan, Beijing University of Posts and Telecommunications（海南女王学院，北京邮电大学）；Department of Computer Science, NUS School of Computing（计算机科学系，国立大学计算机学院）
Towards Category Unification of 3D Single Object Tracking on Point Clouds
School of Electronics and Information, Hangzhou Dianzi University（杭州电子科技大学电子信息学院）；Hanyang University（汉阳大学）；AI Institute, Shanghai Jiao Tong University（上海交通大学AI学院）
Analyzing and Boosting the Power of Fine-Grained Visual Recognition for Multi-modal Large Language Models
Wangxuan Institute of Computer Technology, Peking University（北京大学王选计算机研究所）；School of Intelligence Science and Technology, University of Science and Technology Beijing（北京科技大学智能科学与技术学院）
Cortexica Vision Systems Limited（Cortexica视觉系统有限公司）；Australian Institute for Machine Learning（澳大利亚机器学习研究所）；the University of Adelaide（阿德莱德大学）
In this paper, we present a method for instance ranking and retrieval at fine-grained level based on the global features extracted from a multi-attribute recognition model which is not dependent on landmarks information or part-based annotations. Further, we make this architecture suitable for mobile-device application by adopting the bilinear CNN to make the multi-attribute recognition model smaller (in terms of the number of parameters). The experiments run on the Dress category of DeepFashion In-Shop Clothes Retrieval and CUB200 datasets show that the results of instance retrieval at fine-g…
POSTECH（浦项科技大学）；Graduate School of AI POSTECH（浦项科技大学人工智能研究生院）
Attribute-based person search is the task of finding person images that are best matched with a set of text attributes given as query. The main challenge of this task is the large modality gap between attributes and images. To reduce the gap, we present a new loss for learning cross-modal embeddings in the context of attribute-based person search. We regard a set of attributes as a category of people sharing the same traits. In a joint embedding space of the two modalities, our loss pulls images close to their person categories for modality alignment. More importantly, it pushes apart a pair o…
Object models are gradually progressing from predicting just category labels to providing detailed descriptions of object instances. This motivates the need for large datasets which go beyond traditional object masks and provide richer annotations such as part masks and attributes. Hence, we introduce PACO: Parts and Attributes of Common Objects. It spans 75 object categories, 456 object-part categories and 55 attributes across image (LVIS) and video (Ego4D) datasets. We provide 641K part masks annotated across 260K object boxes, with roughly half of them exhaustively annotated with attributes…
University of Warwick（华威大学）；Aberystwyth University（阿伯里斯特威斯大学）
Due to the costliness of labelled data in real-world applications, semi-supervised object detectors, underpinned by pseudo labelling, are appealing. However, handling confusing samples is nontrivial: discarding valuable confusing samples would compromise the model generalisation while using them for training would exacerbate the confirmation bias issue caused by inevitable mislabelling. To solve this problem, this paper proposes to use confusing samples proactively without label correction. Specifically, a virtual category (VC) is assigned to each confusing sample such that they can safely con…
Dual Prompt Learning for Adapting Vision-Language Models to Downstream Image-Text Retrieval
College of Computer Science, Sichuan University（四川大学计算机学院）；Engineering Research Center of Machine Learning and Industry Intelligence, Ministry of Education, Chengdu, China（教育部机器学习与产业智能工程研究中心）；Deep NeuroCognition Lab, I2R and CFAR, Agency for Science, Technology and Research, Singapore（深度神经认知实验室，I2R和CFAR，科技研究局，新加坡）
Object recognition has become a crucial part of machine learning and computer vision recently. The current approach to object recognition involves Deep Learning and uses Convolutional Neural Networks to learn the pixel patterns of the objects implicitly through backpropagation. However, CNNs require thousands of examples in order to generalize successfully and often require heavy computing resources for training. This is considered rather sluggish when compared to the human ability to generalize and learn new categories given just a single example. Additionally, CNNs make it difficult to expli…
University of Science and Technology of China（中国科学技术大学）；Microsoft Research（微软研究院）
We present variational generative adversarial networks, a general learning framework that combines a variational auto-encoder with a generative adversarial network, for synthesizing images in fine-grained categories, such as faces of a specific person or objects in a category. Our approach models an image as a composition of label and latent attributes in a probabilistic model. By varying the fine-grained category label fed into the resulting generative model, we can generate images in a specific category with randomly drawn values on a latent attribute vector. Our approach has two novel aspec…
Fashion attribute editing is a task that aims to convert the semantic attributes of a given fashion image while preserving the irrelevant regions. Previous works typically employ conditional GANs where the generator explicitly learns the target attributes and directly execute the conversion. These approaches, however, are neither scalable nor generic as they operate only with few limited attributes and a separate generator is required for each dataset or attribute set. Inspired by the recent advancement of diffusion models, we explore the classifier-guided diffusion that leverages the off-the-…
Dissecting Generalized Category Discovery: Multiplex Consensus under Self-Deconstruction
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University（中国教育部多媒体可信感知与高效计算重点实验室，厦门大学）；School of Informatics, Xiamen University（厦门大学信息学院）；Shenzhen University（深圳大学）；The Chinese University of Hong Kong（香港中文大学）；The University of Hong Kong（香港大学）
Attributes are semantically meaningful characteristics whose applicability widely crosses category boundaries. They are particularly important in describing and recognizing concepts where no explicit training example is given, \textit{e.g., zero-shot learning}. Additionally, since attributes are human describable, they can be used for efficient human-computer interaction. In this paper, we propose to employ semantic segmentation to improve facial attribute prediction. The core idea lies in the fact that many facial attributes describe local properties. In other words, the probability of an att…
CASA: Class-Agnostic Shared Attributes in Vision-Language Models for Efficient Incremental Object Detection
School of Electronic and Computer Engineering, Peking University（北京大学电子与计算机工程学院）；Peng Cheng Laboratory（鹏城实验室）；School of Computer Science, Peking University（北京大学计算机学院）
Vehicle re-identification (reID) plays an important role in the automatic analysis of the increasing urban surveillance videos, which has become a hot topic in recent years. However, it poses the critical but challenging problem that is caused by various viewpoints of vehicles, diversified illuminations and complicated environments. Till now, most existing vehicle reID approaches focus on learning metrics or ensemble to derive better representation, which are only take identity labels of vehicle into consideration. However, the attributes of vehicle that contain detailed descriptions are benef…
Key Laboratory of Intelligent Information Processing of Chinese Academy of Sciences (CAS), Institute of Computing Technology, CAS（中国科学院计算技术研究所智能信息处理重点实验室）；University of Chinese Academy of Sciences（中国科学院大学）；CAS Center for Excellence in Brain Science and Intelligence Technology（中国科学院脑科学与智能技术卓越创新中心）；Michigan State University（密歇根州立大学）
Face attribute estimation has many potential applications in video surveillance, face retrieval, and social media. While a number of methods have been proposed for face attribute estimation, most of them did not explicitly consider the attribute correlation and heterogeneity (e.g., ordinal vs. nominal and holistic vs. local) during feature representation learning. In this paper, we present a Deep Multi-Task Learning (DMTL) approach to jointly estimate multiple heterogeneous attributes from a single face image. In DMTL, we tackle attribute correlation and heterogeneity with convolutional neural…
UltrasoundAgents: Hierarchical Multi-Agent Evidence-Chain Reasoning for Breast Ultrasound Diagnosis
Institute of Automation, Chinese Academy of Sciences, Beijing, China（中国科学院自动化研究所，北京，中国）；School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences, Beijing, China（中国科学院大学（大学层面）交叉学科学院，北京，中国）；Centre for Artificial Intelligence and Robotics, Hong Kong Institute of Science & Innovation, Chinese Academy of Sciences, Hong Kong（人工智能与机器人中心，香港创新科学研究院，中国科学院，香港）
Tracking the 6D pose of objects in video sequences is important for robot manipulation. Most prior efforts, however, often assume that the target object's CAD model, at least at a category-level, is available for offline training or during online template matching. This work proposes BundleTrack, a general framework for 6D pose tracking of novel objects, which does not depend upon 3D models, either at the instance or category-level. It leverages the complementary attributes of recent advances in deep learning for segmentation and robust feature extraction, as well as memory-augmented pose grap…
KU Leuven（鲁汶大学）；iMinds（iMinds研究院）；INRIA（法国国家信息与自动化研究所）；CNRS（法国国家科学研究中心）；Univ. Grenoble Alpes（格勒诺布尔阿尔卑斯大学）
Learning the distribution of images in order to generate new samples is a challenging task due to the high dimensionality of the data and the highly non-linear relations that are involved. Nevertheless, some promising results have been reported in the literature recently,building on deep network architectures. In this work, we zoom in on a specific type of image generation: given an image and knowing the category of objects it belongs to (e.g. faces), our goal is to generate a similar and plausible image, but with some altered attributes. This is particularly challenging, as the model needs to…
SPAC: Sampling-based Progressive Attribute Compression for Dense Point Clouds
School of Control Science and Engineering, Shandong University（山东大学控制科学与工程学院）；School of Engineering and Sustainable Development, De Montfort University（德蒙福特大学工程与可持续发展学院）；School of Data Science, Lingnan University（岭南大学数据科学学院）
Being able to forecast the popularity of new garment designs is very important in an industry as fast paced as fashion, both in terms of profitability and reducing the problem of unsold inventory. Here, we attempt to address this task in order to provide informative forecasts to fashion designers within a virtual reality designer application that will allow them to fine tune their creations based on current consumer preferences within an interactive and immersive environment. To achieve this we have to deal with the following central challenges: (1) the proposed method should not hinder the cr…
