In this paper, we consider a highly general image recognition setting wherein, given a labelled and unlabelled set of images, the task is to categorize all images in the unlabelled set. Here, the unlabelled images may come from labelled classes or from novel ones. Existing recognition methods are not able to deal with this setting, because they make several restrictive assumptions, such as the unlabelled instances only coming from known - or unknown - classes, and the number of unknown classes being known a-priori. We address the more unconstrained setting, naming it 'Generalized Category Discovery', and challenge all these assumptions. We first establish strong baselines by taking state-of-the-art algorithms from novel category discovery and adapting them for this task. Next, we propose the use of vision transformers with contrastive representation learning for this open-world setting. We then introduce a simple yet effective semi-supervised $k$-means method to cluster the unlabelled data into seen and unseen classes automatically, substantially outperforming the baselines. Finally, we also propose a new approach to estimate the number of classes in the unlabelled data. We thoroughly evaluate our approach on public datasets for generic object classification and on fine-grained datasets, leveraging the recent Semantic Shift Benchmark suite. Project page at https://www.robots.ox.ac.uk/~vgg/research/gcd
Although a number of studies are devoted to novel category discovery, most of them assume a static setting where both labeled and unlabeled data are given at once for finding new categories. In this work, we focus on the application scenarios where unlabeled data are continuously fed into the category discovery system. We refer to it as the {\bf Continuous Category Discovery} ({\bf CCD}) problem, which is significantly more challenging than the static setting. A common challenge faced by novel category discovery is that different sets of features are needed for classification and category disc…
Hierarchical Generalized Category Discovery for Brain Tumor Classification in Digital Pathology
Institute of Clinical Epidemiology, Public Health, Health Economics, Medical Statistics and Informatics（临床流行病学、公共卫生、健康经济学、医学统计与信息学研究所）；Institute of Neuropathology and Neuromolecular Pathology（神经病理学与神经分子病理学研究所）
Xi’an Jiaotong University（西安交通大学）；University of Massachusetts Boston（马萨诸塞大学波士顿分校）；Zhejiang University（浙江大学）；Lenovo Research（联想研究院）
Novel category discovery aims at adapting models trained on known categories to novel categories. Previous works only focus on the scenario where known and novel categories are of the same granularity. In this paper, we investigate a new practical scenario called Fine-grained Category Discovery under Coarse-grained supervision (FCDC). FCDC aims at discovering fine-grained categories with only coarse-grained labeled data, which can adapt models to categories of different granularity from known ones and reduce significant labeling cost. It is also a challenging task since supervised training on …
Comparing Euclidean and Hyperbolic K-Means for Generalized Category Discovery
Visual Analysis and Perception Lab, Aalborg University, Aalborg, Denmark（奥尔堡大学视觉分析与感知实验室）；Pioneer Centre for AI, Denmark（丹麦先锋人工智能中心）；Center for Software Technology, University of Southern Denmark, Vejle, Denmark（丹麦南部大学软件技术中心）
Generalized Category Discovery with Large Language Models in the Loop
School of Automation Science and Engineering, Xi’an Jiaotong University（西安交通大学自动化科学与工程学院）；School of Computer Science and Technology, Xi’an Jiaotong University（西安交通大学计算机科学与技术学院）；Lenovo Research（联想研究院）；University of Massachusetts Boston（马萨诸塞大学波士顿分校）
ProtoGCD: Unified and Unbiased Prototype Learning for Generalized Category Discovery
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Centre for Artificial Intelligence and Robotics, Hong Kong Institute of Science and Innovation, Chinese Academy of Sciences（中国科学院香港创新研究院人工智能与机器人中心）
Category Adaptation Meets Projected Distillation in Generalized Continual Category Discovery
IDEAS NCBR；Warsaw University of Technology（华沙理工大学）；Gdańsk University of Technology（格但斯克理工大学）；Tooploox；Autonomous University of Barcelona（巴塞罗那自治大学）；Computer Vision Center（计算机视觉中心）
ClearGCD: Mitigating Shortcut Learning For Robust Generalized Category Discovery
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Loughborough University（洛桑大学）
Georgia Institute of Technology（佐治亚理工学院）；Georgia Tech Research Institute（佐治亚理工研究所）
Automatically discovering image categories in unlabeled natural images is one of the important goals of unsupervised learning. However, the task is challenging and even human beings define visual categories based on a large amount of prior knowledge. In this paper, we similarly utilize prior knowledge to facilitate the discovery of image categories. We present a novel end-to-end network to map unlabeled images to categories as a clustering network. We propose that this network can be learned with contrastive loss which is only based on weak binary pair-wise constraints. Such binary constraints…
School of Computer Science and Engineering, Southeast University（东南大学计算机科学与工程学院）；State Key Laboratory of Media Convergence and Communication, Communication University of China（中国传媒大学媒介融合与传播国家重点实验室）；Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education（教育部新一代人工智能技术及其交叉应用重点实验室（东南大学））
Georgia Institute of Technology（佐治亚理工学院）；Georgia Tech Research Institute（佐治亚理工研究院）
Neural network-based clustering has recently gained popularity, and in particular a constrained clustering formulation has been proposed to perform transfer learning and image category discovery using deep learning. The core idea is to formulate a clustering objective with pairwise constraints that can be used to train a deep clustering network; therefore the cluster assignments and their underlying feature representations are jointly optimized end-to-end. In this work, we provide a novel clustering formulation to address scalability issues of previous work in terms of optimizing deeper networ…
Dissecting Generalized Category Discovery: Multiplex Consensus under Self-Deconstruction
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University（中国教育部多媒体可信感知与高效计算重点实验室，厦门大学）；School of Informatics, Xiamen University（厦门大学信息学院）；Shenzhen University（深圳大学）；The Chinese University of Hong Kong（香港中文大学）；The University of Hong Kong（香港大学）
Sun Yat-Sen University（中山大学）；Guangdong University of Technology（广东工业大学）
As a promising solution of reducing annotation cost, training multi-label models with partial positive labels (MLR-PPL), in which merely few positive labels are known while other are missing, attracts increasing attention. Due to the absence of any negative labels, previous works regard unknown labels as negative and adopt traditional MLR algorithms. To reject noisy labels, recent works regard large loss samples as noise but ignore the semantic correlation different multi-label images. In this work, we propose to explore semantic correlation among different images to facilitate the MLR-PPL tas…
Tongji University（同济大学）；University of Edinburgh（爱丁堡大学）；LunarAI
We address the problem of generalized category discovery (GCD) in this paper, i.e. clustering the unlabeled images leveraging the information from a set of seen classes, where the unlabeled images could contain both seen classes and unseen classes. The seen classes can be seen as an implicit criterion of classes, which makes this setting different from unsupervised clustering where the cluster criteria may be ambiguous. We mainly concern the problem of discovering categories within a fine-grained dataset since it is one of the most direct applications of category discovery, i.e. helping expert…
Columbia University（哥伦比亚大学）；Brandeis University（布兰迪斯大学）
Generalized Zero-Shot Learning (GZSL) and Open-Set Recognition (OSR) are two mainstream settings that greatly extend conventional visual object recognition. However, the limitations of their problem settings are not negligible. The novel categories in GZSL require pre-defined semantic labels, making the problem setting less realistic; the oversimplified unknown class in OSR fails to explore the innate fine-grained and mixed structures of novel categories. In light of this, we are motivated to consider a new problem setting named Zero-Knowledge Zero-Shot Learning (ZK-ZSL) that assumes no prior …
Consistent Supervised-Unsupervised Alignment for Generalized Category Discovery
National Key Laboratory of Human-Machine Hybrid Augmented Intelligence, National Engineering Research Center for Visual Information and Applications, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University（国家人类-机器混合增强智能重点实验室、视觉信息与应用国家工程研究中心、人工智能与机器人研究院、西安交通大学）；School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen（计算机科学与技术学院、哈尔滨工业大学深圳校区）；School of Software Engineering, Xi’an Jiaotong University（软件工程学院、西安交通大学）；Kuaishou Technology（快手科技）；Faculty of Microelectronics, Shenzhen University of Advanced Technology（微电子学院、深圳先进技术大学）
Sparsity Hurts: Simple Linear Adapter Can Boost Generalized Category Discovery
School of Computer Science and Engineering, Southeast University（东南大学计算机科学与工程学院）；Key Laboratory of Computer Network and Information Integration (Southeast University), Ministry of Education, China（计算机网络与信息集成重点实验室（东南大学），教育部，中国）
Visual AI Lab, School of Computing and Data Science, The University of Hong Kong, Hong Kong（视觉人工智能实验室，计算与数据科学学院，香港大学，香港）；Visual AI Lab, School of Computing（视觉人工智能实验室，计算学院）；Data Science, The University of Hong Kong, Hong Kong（数据科学，香港大学，香港）
Google（谷歌公司）；University of Bristol（布里斯托尔大学）；The University of Hong Kong（香港大学）
This paper studies the problem of novel category discovery on single- and multi-modal data with labels from different but relevant categories. We present a generic, end-to-end framework to jointly learn a reliable representation and assign clusters to unlabelled data. To avoid over-fitting the learnt embedding to labelled data, we take inspiration from self-supervised representation learning by noise-contrastive estimation and extend it to jointly handle labelled and unlabelled data. In particular, we propose using category discrimination on labelled data and cross-modal discrimination on mult…
School of Computer Science and Engineering, Nanjing University of Science and Technology, China（南京理工大学计算机科学与工程学院）；School of Engineering, Newcastle University, Newcastle upon Tyne, United Kingdom（新castle大学工程学院）
Generalized Category Discovery in Hyperspectral Images via Prototype Subspace Modeling
Department of Telecommunications and Information Processing, Ghent University, Belgium（电信与信息处理系，根特大学，比利时）；Royal Institute for Cultural Heritage (KIK-IRPA), Brussels, Belgium（皇家文化遗产研究院（KIK-IRPA），布鲁塞尔，比利时）；School of Computer Science, China University of Geosciences, Wuhan, China（计算机科学学院，中国地质大学（武汉），中国）
Explainable Novel Category Discovery in Semantic Concept Space
Department of Computer Science, University of South Dakota（南达科他大学计算机科学系）；Department of Computer Science and Software Engineering, Auburn University（奥本大学计算机科学与软件工程系）
Component Adaptive Clustering for Generalized Category Discovery
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Shenzhen University of Advanced Technology（深圳先进技术大学）
Tongji University（同济大学）；The University of Hong Kong（香港大学）；Google Research（谷歌研究院）；University of Bristol（布里斯托大学）
In this paper, we tackle the problem of novel visual category discovery, i.e., grouping unlabelled images from new classes into different semantic partitions by leveraging a labelled dataset that contains images from other different but relevant categories. This is a more realistic and challenging setting than conventional semi-supervised learning. We propose a two-branch learning framework for this problem, with one branch focusing on local part-level information and the other branch focusing on overall characteristics. To transfer knowledge from the labelled data to the unlabelled, we propos…
