This paper studies the problem of novel category discovery on single- and multi-modal data with labels from different but relevant categories. We present a generic, end-to-end framework to jointly learn a reliable representation and assign clusters to unlabelled data. To avoid over-fitting the learnt embedding to labelled data, we take inspiration from self-supervised representation learning by noise-contrastive estimation and extend it to jointly handle labelled and unlabelled data. In particular, we propose using category discrimination on labelled data and cross-modal discrimination on multi-modal data to augment instance discrimination used in conventional contrastive learning approaches. We further employ Winner-Take-All (WTA) hashing algorithm on the shared representation space to generate pairwise pseudo labels for unlabelled data to better predict cluster assignments. We thoroughly evaluate our framework on large-scale multi-modal video benchmarks Kinetics-400 and VGG-Sound, and image benchmarks CIFAR10, CIFAR100 and ImageNet, obtaining state-of-the-art results.
School of Computer Science and Engineering, Southeast University（东南大学计算机科学与工程学院）；State Key Laboratory of Media Convergence and Communication, Communication University of China（中国传媒大学媒介融合与传播国家重点实验室）；Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education（教育部新一代人工智能技术及其交叉应用重点实验室（东南大学））
Multi-Modal Representation Learning via Semi-Supervised Rate Reduction for Generalized Category Discovery
School of Artificial Intelligence, Beijing University of Posts and Telecommunications（北京邮电大学人工智能学院）；Intellifusion Inc., Shenzhen, P.R. China（深圳云天励飞技术股份有限公司）
Sun Yat-Sen University（中山大学）；Guangdong University of Technology（广东工业大学）
As a promising solution of reducing annotation cost, training multi-label models with partial positive labels (MLR-PPL), in which merely few positive labels are known while other are missing, attracts increasing attention. Due to the absence of any negative labels, previous works regard unknown labels as negative and adopt traditional MLR algorithms. To reject noisy labels, recent works regard large loss samples as noise but ignore the semantic correlation different multi-label images. In this work, we propose to explore semantic correlation among different images to facilitate the MLR-PPL tas…
Tongji University（同济大学）；University of Edinburgh（爱丁堡大学）；LunarAI
We address the problem of generalized category discovery (GCD) in this paper, i.e. clustering the unlabeled images leveraging the information from a set of seen classes, where the unlabeled images could contain both seen classes and unseen classes. The seen classes can be seen as an implicit criterion of classes, which makes this setting different from unsupervised clustering where the cluster criteria may be ambiguous. We mainly concern the problem of discovering categories within a fine-grained dataset since it is one of the most direct applications of category discovery, i.e. helping expert…
Consistent Supervised-Unsupervised Alignment for Generalized Category Discovery
National Key Laboratory of Human-Machine Hybrid Augmented Intelligence, National Engineering Research Center for Visual Information and Applications, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University（国家人类-机器混合增强智能重点实验室、视觉信息与应用国家工程研究中心、人工智能与机器人研究院、西安交通大学）；School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen（计算机科学与技术学院、哈尔滨工业大学深圳校区）；School of Software Engineering, Xi’an Jiaotong University（软件工程学院、西安交通大学）；Kuaishou Technology（快手科技）；Faculty of Microelectronics, Shenzhen University of Advanced Technology（微电子学院、深圳先进技术大学）
Visual AI Lab, School of Computing and Data Science, The University of Hong Kong, Hong Kong（视觉人工智能实验室，计算与数据科学学院，香港大学，香港）；Visual AI Lab, School of Computing（视觉人工智能实验室，计算学院）；Data Science, The University of Hong Kong, Hong Kong（数据科学，香港大学，香港）
School of Computer Science and Engineering, Nanjing University of Science and Technology, China（南京理工大学计算机科学与工程学院）；School of Engineering, Newcastle University, Newcastle upon Tyne, United Kingdom（新castle大学工程学院）
Technical University of Munich（慕尼黑工业大学）；Toyota Motor Europe（丰田汽车欧洲公司）
Object pose estimation is crucial for robotic applications and augmented reality. Beyond instance level 6D object pose estimation methods, estimating category-level pose and shape has become a promising trend. As such, a new research field needs to be supported by well-designed datasets. To provide a benchmark with high-quality ground truth annotations to the community, we introduce a multimodal dataset for category-level object pose estimation with photometrically challenging objects termed PhoCaL. PhoCaL comprises 60 high quality 3D models of household objects over 8 categories including hig…
Generalized Category Discovery in Hyperspectral Images via Prototype Subspace Modeling
Department of Telecommunications and Information Processing, Ghent University, Belgium（电信与信息处理系，根特大学，比利时）；Royal Institute for Cultural Heritage (KIK-IRPA), Brussels, Belgium（皇家文化遗产研究院（KIK-IRPA），布鲁塞尔，比利时）；School of Computer Science, China University of Geosciences, Wuhan, China（计算机科学学院，中国地质大学（武汉），中国）
Component Adaptive Clustering for Generalized Category Discovery
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Shenzhen University of Advanced Technology（深圳先进技术大学）
Tongji University（同济大学）；The University of Hong Kong（香港大学）；Google Research（谷歌研究院）；University of Bristol（布里斯托大学）
In this paper, we tackle the problem of novel visual category discovery, i.e., grouping unlabelled images from new classes into different semantic partitions by leveraging a labelled dataset that contains images from other different but relevant categories. This is a more realistic and challenging setting than conventional semi-supervised learning. We propose a two-branch learning framework for this problem, with one branch focusing on local part-level information and the other branch focusing on overall characteristics. To transfer knowledge from the labelled data to the unlabelled, we propos…
Synergistic Dual-Branch Adaptation for Multi-modal Generalized Category Discovery
College of Computer Science, Nankai University（南开大学计算机学院）；State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所多模态人工智能系统国家重点实验室）
DIG-FACE: De-biased Learning for Generalized Facial Expression Category Discovery
University of Cambridge（剑桥大学）；Baidu Inc（百度公司）；Central China Normal University（华中师范大学）；La Trobe University（拉筹伯大学）；China University of Geosciences（中国地质大学）；JD Explore Academy（京东探索研究院）
Generalized Fine-Grained Category Discovery with Multi-Granularity Conceptual Experts
Department of Information Engineering and Computer Science, University of Trento（特伦托大学信息工程与计算机科学系）；School of Computer Science and Information Engineering, Hefei University of Technology（合肥工业大学计算机科学与信息工程学院）
National Institute of Informatics（国立信息学研究所）；The University of Electro-Communications（电气通信大学）；The University of Tokyo（东京大学）
In this work, travel destination and business location are taken as venues. Discovering a venue by a photo is very important for context-aware applications. Unfortunately, few efforts paid attention to complicated real images such as venue photos generated by users. Our goal is fine-grained venue discovery from heterogeneous social multimodal data. To this end, we propose a novel deep learning model, Category-based Deep Canonical Correlation Analysis (C-DCCA). Given a photo as input, this model performs (i) exact venue search (find the venue where the photo was taken), and (ii) group venue sea…
InfoSculpt: Sculpting the Latent Space for Generalized Category Discovery
College of Intelligent Robotics and Advance Manufacturing, Fudan University（复旦大学智能机器人与先进制造学院）；School of Microelectronics, Fudan University（复旦大学微电子学院）；Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；School of Information Science and Engineering, East China University of Science and Technology（东华大学信息科学与工程学院）
Prior-Constrained Association Learning for Fine-Grained Generalized Category Discovery
School of Computer and Electronic Information, Nanjing Normal University（南京师范大学计算机与电子信息学院）；School of Computer Science and Information Engineering, Hefei University of Technology（合肥工业大学计算机与信息工程学院）；College of Information Science and Electronic Engineering, Zhejiang University（浙江大学信息科学与电子工程学院）
Category Adaptation Meets Projected Distillation in Generalized Continual Category Discovery
IDEAS NCBR；Warsaw University of Technology（华沙理工大学）；Gdańsk University of Technology（格但斯克理工大学）；Tooploox；Autonomous University of Barcelona（巴塞罗那自治大学）；Computer Vision Center（计算机视觉中心）
Sun Yat-Sen University（中山大学）；SYSU-CMU Shunde International Joint Research Institute（中山大学-卡内基梅隆大学顺德国际联合研究院）
This paper investigates a general framework to discover categories of unlabeled scene images according to their appearances (i.e., textures and structures). We jointly solve the two coupled tasks in an unsupervised manner: (i) classifying images without pre-determining the number of categories, and (ii) pursuing generative model for each category. In our method, each image is represented by two types of image descriptors that are effective to capture image appearances from different aspects. By treating each image as a graph vertex, we build up an graph, and pose the image categorization as a …
Material attributes have been shown to provide a discriminative intermediate representation for recognizing materials, especially for the challenging task of recognition from local material appearance (i.e., regardless of object and scene context). In the past, however, material attributes have been recognized separately preceding category recognition. In contrast, neuroscience studies on material perception and computer vision research on object and place recognition have shown that attributes are produced as a by-product during the category recognition process. Does the same hold true for ma…
Multi-modal pre-training and knowledge discovery are two important research topics in multi-modal machine learning. Nevertheless, none of existing works make attempts to link knowledge discovery with knowledge guided multi-modal pre-training. In this paper, we propose to unify them into a continuous learning framework for mutual improvement. Taking the open-domain uni-modal datasets of images and texts as input, we maintain a knowledge graph as the foundation to support these two tasks. For knowledge discovery, a pre-trained model is used to identify cross-modal links on the graph. For model p…
Although a number of studies are devoted to novel category discovery, most of them assume a static setting where both labeled and unlabeled data are given at once for finding new categories. In this work, we focus on the application scenarios where unlabeled data are continuously fed into the category discovery system. We refer to it as the {\bf Continuous Category Discovery} ({\bf CCD}) problem, which is significantly more challenging than the static setting. A common challenge faced by novel category discovery is that different sets of features are needed for classification and category disc…
Towards Distribution-Agnostic Generalized Category Discovery
Zhejiang University（浙江大学）；The Hong Kong University of Science and Technology（香港科技大学）；Centre for Frontier AI Research, A*STAR（新加坡科技研究局前沿人工智能研究中心）；Angelalign Technology Inc.（时代天使科技有限公司）
TALON: Test-time Adaptive Learning for On-the-Fly Category Discovery
College of Information and Electrical Engineering（信息与电气工程学院）；China Agricultural University（中国农业大学）；Department of Electrical and Computer Engineering（电气与计算机工程系）；University of Toronto（多伦多大学）；Institute of Brain-Inspired Intelligence and Artificial Intelligence（脑启发智能与人工智能研究院）；Fudan University（复旦大学）；School of Computer and Information Technology（计算机与信息学院）；Beijing Jiaotong University（北京交通大学）；Department of Computer Science and Software Engineering（计算机科学与软件工程系）；Concordia University（Concordia大学）
