Generalized Category Discovery (GCD) is a challenging task in which, given a partially labelled dataset, models must categorize all unlabelled instances, regardless of whether they come from labelled categories or from new ones. In this paper, we challenge a remaining assumption in this task: that all images share the same domain. Specifically, we introduce a new task and method to handle GCD when the unlabelled data also contains images from different domains to the labelled set. Our proposed `HiLo' networks extract High-level semantic and Low-level domain features, before minimizing the mutual information between the representations. Our intuition is that the clusterings based on domain information and semantic information should be independent. We further extend our method with a specialized domain augmentation tailored for the GCD task, as well as a curriculum learning approach. Finally, we construct a benchmark from corrupted fine-grained datasets as well as a large-scale evaluation on DomainNet with real-world domain shifts, reimplementing a number of GCD baselines in this setting. We demonstrate that HiLo outperforms SoTA category discovery models by a large margin on all evaluations.
Category Adaptation Meets Projected Distillation in Generalized Continual Category Discovery
IDEAS NCBR；Warsaw University of Technology（华沙理工大学）；Gdańsk University of Technology（格但斯克理工大学）；Tooploox；Autonomous University of Barcelona（巴塞罗那自治大学）；Computer Vision Center（计算机视觉中心）
Tongji University（同济大学）；The University of Hong Kong（香港大学）；Google Research（谷歌研究院）；University of Bristol（布里斯托大学）
In this paper, we tackle the problem of novel visual category discovery, i.e., grouping unlabelled images from new classes into different semantic partitions by leveraging a labelled dataset that contains images from other different but relevant categories. This is a more realistic and challenging setting than conventional semi-supervised learning. We propose a two-branch learning framework for this problem, with one branch focusing on local part-level information and the other branch focusing on overall characteristics. To transfer knowledge from the labelled data to the unlabelled, we propos…
National Institute of Informatics（国立信息学研究所）；The University of Electro-Communications（电气通信大学）；The University of Tokyo（东京大学）
In this work, travel destination and business location are taken as venues. Discovering a venue by a photo is very important for context-aware applications. Unfortunately, few efforts paid attention to complicated real images such as venue photos generated by users. Our goal is fine-grained venue discovery from heterogeneous social multimodal data. To this end, we propose a novel deep learning model, Category-based Deep Canonical Correlation Analysis (C-DCCA). Given a photo as input, this model performs (i) exact venue search (find the venue where the photo was taken), and (ii) group venue sea…
Prior-Constrained Association Learning for Fine-Grained Generalized Category Discovery
School of Computer and Electronic Information, Nanjing Normal University（南京师范大学计算机与电子信息学院）；School of Computer Science and Information Engineering, Hefei University of Technology（合肥工业大学计算机与信息工程学院）；College of Information Science and Electronic Engineering, Zhejiang University（浙江大学信息科学与电子工程学院）
Unlocking Optical Prior: Spectrum-Guided Knowledge Transfer for SAR Generalized Category Discovery
College of Electronic Science and Technology, National University of Defense Technology（电子科学与技术学院，国防科技大学）；State Key Laboratory of Complex System Simulation and Modeling Technology（复杂系统仿真与建模技术国家重点实验室）
Generalized Category Discovery in Federated Graph Learning
Shandong University, School of Airspace Science and Engineering（山东大学，空天学院）；Beijing Institute of Technology, School of Computer Science and Technology（北京理工大学，计算机科学与技术学院）
University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；Peking University（北京大学）；Beijing Institute for General Artificial Intelligence（北京通用人工智能研究院）；Tsinghua University（清华大学）
Understanding what objects could furnish for humans-namely, learning object affordance-is the crux to bridge perception and action. In the vision community, prior work primarily focuses on learning object affordance with dense (e.g., at a per-pixel level) supervision. In stark contrast, we humans learn the object affordance without dense labels. As such, the fundamental question to devise a computational model is: What is the natural way to learn the object affordance from visual appearance and geometry with humanlike sparse supervision? In this work, we present a new task of part-level afford…
Understanding objects in terms of their individual parts is important, because it enables a precise understanding of the objects' geometrical structure, and enhances object recognition when the object is seen in a novel pose or under partial occlusion. However, the manual annotation of parts in large scale datasets is time consuming and expensive. In this paper, we aim at discovering object parts in an unsupervised manner, i.e., without ground-truth part or keypoint annotations. Our approach builds on the intuition that objects of the same class in a similar pose should have their parts aligne…
Tufts University（塔夫茨大学）；Austrian Institute of Technology（奥地利技术研究所）
In order for artificial agents to successfully perform tasks in changing environments, they must be able to both detect and adapt to novelty. However, visual novelty detection research often only evaluates on repurposed datasets such as CIFAR-10 originally intended for object classification, where images focus on one distinct, well-centered object. New benchmarks are needed to represent the challenges of navigating the complex scenes of an open world. Our new NovelCraft dataset contains multimodal episodic data of the images and symbolic world-states seen by an agent completing a pogo stick as…
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences（中国科学院脑科学与智能技术卓越创新中心）；University of California at Merced（加州大学默塞德分校）
This paper tackles the problem of novel category discovery (NCD), which aims to discriminate unknown categories in large-scale image collections. The NCD task is challenging due to the closeness to the real-world scenarios, where we have only encountered some partial classes and images. Unlike other works on the NCD, we leverage the prototypes to emphasize the importance of category discrimination and alleviate the issue of missing annotations of novel classes. Concretely, we propose a novel adaptive prototype learning method consisting of two main stages: prototypical representation learning …
Explainable Novel Category Discovery in Semantic Concept Space
Department of Computer Science, University of South Dakota（南达科他大学计算机科学系）；Department of Computer Science and Software Engineering, Auburn University（奥本大学计算机科学与软件工程系）
Learning Like Humans: Analogical Concept Learning for Generalized Category Discovery
State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University（人机混合增强智能国家重点实验室，人工智能与机器人研究院，西安交通大学）；School of Software Engineering, Xi’an Jiaotong University（软件工程学院，西安交通大学）；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；Shenzhen University of Advanced Technology（深圳大学）
Proxy Anchor-based Unsupervised Learning for Continuous Generalized Category Discovery
Korea Advanced Institute of Science and Technology（韩国科学技术院）；Samsung Electro-Mechanics（三星电机）；German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）；RPTU Kaiserslautern-Landau（凯泽斯劳滕-兰道大学）
Georgia Institute of Technology（佐治亚理工学院）；Georgia Tech Research Institute（佐治亚理工研究院）
Neural network-based clustering has recently gained popularity, and in particular a constrained clustering formulation has been proposed to perform transfer learning and image category discovery using deep learning. The core idea is to formulate a clustering objective with pairwise constraints that can be used to train a deep clustering network; therefore the cluster assignments and their underlying feature representations are jointly optimized end-to-end. In this work, we provide a novel clustering formulation to address scalability issues of previous work in terms of optimizing deeper networ…
School of Computer Science and Engineering, Southeast University（东南大学计算机科学与工程学院）；State Key Laboratory of Media Convergence and Communication, Communication University of China（中国传媒大学媒介融合与传播国家重点实验室）；Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education（教育部新一代人工智能技术及其交叉应用重点实验室（东南大学））
National Engineering Laboratory for Big Data Analytics（大数据分析国家工程实验室）；Department of Computer Science, University of Massachusetts Boston（马萨诸塞大学波士顿分校计算机科学系）；Lenovo Research（联想研究院）；Department of Engineering, University of Massachusetts Boston（马萨诸塞大学波士顿分校工程系）
Generalized Category Discovery (GCD) aims to recognize both known and novel categories from a set of unlabeled data, based on another dataset labeled with only known categories. Without considering differences between known and novel categories, current methods learn about them in a coupled manner, which can hurt model's generalization and discriminative ability. Furthermore, the coupled training approach prevents these models transferring category-specific knowledge explicitly from labeled data to unlabeled data, which can lose high-level semantic information and impair model performance. To …
Sun Yat-Sen University（中山大学）；Guangdong University of Technology（广东工业大学）
As a promising solution of reducing annotation cost, training multi-label models with partial positive labels (MLR-PPL), in which merely few positive labels are known while other are missing, attracts increasing attention. Due to the absence of any negative labels, previous works regard unknown labels as negative and adopt traditional MLR algorithms. To reject noisy labels, recent works regard large loss samples as noise but ignore the semantic correlation different multi-label images. In this work, we propose to explore semantic correlation among different images to facilitate the MLR-PPL tas…
Tongji University（同济大学）；University of Edinburgh（爱丁堡大学）；LunarAI
We address the problem of generalized category discovery (GCD) in this paper, i.e. clustering the unlabeled images leveraging the information from a set of seen classes, where the unlabeled images could contain both seen classes and unseen classes. The seen classes can be seen as an implicit criterion of classes, which makes this setting different from unsupervised clustering where the cluster criteria may be ambiguous. We mainly concern the problem of discovering categories within a fine-grained dataset since it is one of the most direct applications of category discovery, i.e. helping expert…
Consistent Supervised-Unsupervised Alignment for Generalized Category Discovery
National Key Laboratory of Human-Machine Hybrid Augmented Intelligence, National Engineering Research Center for Visual Information and Applications, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University（国家人类-机器混合增强智能重点实验室、视觉信息与应用国家工程研究中心、人工智能与机器人研究院、西安交通大学）；School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen（计算机科学与技术学院、哈尔滨工业大学深圳校区）；School of Software Engineering, Xi’an Jiaotong University（软件工程学院、西安交通大学）；Kuaishou Technology（快手科技）；Faculty of Microelectronics, Shenzhen University of Advanced Technology（微电子学院、深圳先进技术大学）
Visual AI Lab, School of Computing and Data Science, The University of Hong Kong, Hong Kong（视觉人工智能实验室，计算与数据科学学院，香港大学，香港）；Visual AI Lab, School of Computing（视觉人工智能实验室，计算学院）；Data Science, The University of Hong Kong, Hong Kong（数据科学，香港大学，香港）
Google（谷歌公司）；University of Bristol（布里斯托尔大学）；The University of Hong Kong（香港大学）
This paper studies the problem of novel category discovery on single- and multi-modal data with labels from different but relevant categories. We present a generic, end-to-end framework to jointly learn a reliable representation and assign clusters to unlabelled data. To avoid over-fitting the learnt embedding to labelled data, we take inspiration from self-supervised representation learning by noise-contrastive estimation and extend it to jointly handle labelled and unlabelled data. In particular, we propose using category discrimination on labelled data and cross-modal discrimination on mult…
