Beyond Simple Meta-Learning: Multi-Purpose Models for Multi-Domain, Active and Continual Few-Shot Learning
Beam AI；University of British Columbia（不列颠哥伦比亚大学）；MILA（蒙特利尔学习算法研究所）；Vector Institute（向量研究所）；University of Amsterdam（阿姆斯特丹大学）；Inverted AI；Northeastern University（东北大学）；Google Research（谷歌研究院）；CIFAR（加拿大高等研究院）；Amazon（亚马逊）
Peyman Bateni, Jarred Barber, Raghav Goyal, Vaden Masrani, Jan-Willem van de Meent, Leonid Sigal, Frank Wood
Modern deep learning requires large-scale extensively labelled datasets for training. Few-shot learning aims to alleviate this issue by learning effectively from few labelled examples. In previously proposed few-shot visual classifiers, it is assumed that the feature manifold, where classifier decisions are made, has uncorrelated feature dimensions and uniform feature variance. In this work, we focus on addressing the limitations arising from this assumption by proposing a variance-sensitive class of models that operates in a low-label regime. The first method, Simple CNAPS, employs a hierarchically regularized Mahalanobis-distance based classifier combined with a state of the art neural adaptive feature extractor to achieve strong performance on Meta-Dataset, mini-ImageNet and tiered-ImageNet benchmarks. We further extend this approach to a transductive learning setting, proposing Transductive CNAPS. This transductive method combines a soft k-means parameter refinement procedure with a two-step task encoder to achieve improved test-time classification accuracy using unlabelled data. Transductive CNAPS achieves state of the art performance on Meta-Dataset. Finally, we explore the use of our methods (Simple and Transductive) for "out of the box" continual and active learning. Extensive experiments on large scale benchmarks illustrate robustness and versatility of this, relatively speaking, simple class of models. All trained model checkpoints and corresponding source codes have been made publicly available.
Computer Vision Center, Autonomous University of Barcelona (UAB)（巴塞罗那自治大学计算机视觉中心）；Computer Vision and Machine Learning, University of Exeter（埃克塞特大学计算机视觉与机器学习）
We propose bookworm continual learning(BCL), a flexible setting where unseen classes can be inferred via a semantic model, and the visual model can be updated continually. Thus BCL generalizes both continual learning (CL) and zero-shot learning (ZSL). We also propose the bidirectional imagination (BImag) framework to address BCL where features of both past and future classes are generated. We observe that conditioning the feature generator on attributes can actually harm the continual learning ability, and propose two variants (joint class-attribute conditioning and asymmetric generation) to a…
Continual zero-shot learning(CZSL) is a new domain to classify objects sequentially the model has not seen during training. It is more suitable than zero-shot and continual learning approaches in real-case scenarios when data may come continually with only attributes for a few classes and attributes and features for other classes. Continual learning(CL) suffers from catastrophic forgetting, and zero-shot learning(ZSL) models cannot classify objects like state-of-the-art supervised classifiers due to lack of actual data(or features) during training. This paper proposes a novel continual zero-sh…
Indian Institute of Science（印度科学学院）；IIT Madras（印度理工学院马德拉斯分校）
Recently, zero-shot learning (ZSL) emerged as an exciting topic and attracted a lot of attention. ZSL aims to classify unseen classes by transferring the knowledge from seen classes to unseen classes based on the class description. Despite showing promising performance, ZSL approaches assume that the training samples from all seen classes are available during the training, which is practically not feasible. To address this issue, we propose a more generalized and practical setup for ZSL, i.e., continual ZSL (CZSL), where classes arrive sequentially in the form of a task and it actively learns …
Both few-shot and continual learning have seen substantial progress in the last years due to the introduction of proper benchmarks. That being said, the field has still to frame a suite of benchmarks for the highly desirable setting of continual few-shot learning, where the learner is presented a number of few-shot tasks, one after the other, and then asked to perform well on a validation set stemming from all previously seen tasks. Continual few-shot learning has a small computational footprint and is thus an excellent setting for efficient investigation and experimentation. In this paper we …
Continual Learning in Open-vocabulary Classification with Complementary Memory Systems
Siebel School of Computing and Data Science（西贝尔计算与数据科学学院）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of California Merced（加州大学默塞德分校）
Online few-shot learning describes a setting where models are trained and evaluated on a stream of data while learning emerging classes. While prior work in this setting has achieved very promising performance on instance classification when learning from data-streams composed of a single indoor environment, we propose to extend this setting to consider object classification on a series of several indoor environments, which is likely to occur in applications such as robotics. Importantly, our setting, which we refer to as online few-shot continual learning, injects the well-studied issue of ca…
University of Tübingen（图宾根大学）；MPI for Intelligent Systems（马克斯·普朗克智能系统研究所）
Large pre-trained, zero-shot capable models have shown considerable success both for standard transfer and adaptation tasks, with particular robustness towards distribution shifts. In addition, subsequent fine-tuning can considerably improve performance on a selected downstream task. However, through naive fine-tuning, these zero-shot models lose their generalizability and robustness towards distribution shifts. This is a particular problem for tasks such as Continual Learning (CL), where continuous adaptation has to be performed as new task distributions are introduced sequentially. In this w…
University of Oulu（奥卢大学）；Nanyang Technological University（南洋理工大学）；ByteDance（字节跳动）；National University of Defense Technology（国防科技大学）
The goal of Few-Shot Continual Learning (FSCL) is to incrementally learn novel tasks with limited labeled samples and preserve previous capabilities simultaneously, while current FSCL methods are all for the class-incremental purpose. Moreover, the evaluation of FSCL solutions is only the cumulative performance of all encountered tasks, but there is no work on exploring the domain generalization ability. Domain generalization is a challenging yet practical task that aims to generalize beyond training domains. In this paper, we set up a Generalized FSCL (GFSCL) protocol involving both class- an…
Most of the existing artificial neural networks(ANNs) fail to learn continually due to catastrophic forgetting, while humans can do the same by maintaining previous tasks' performances. Although storing all the previous data can alleviate the problem, it takes a large memory, infeasible in real-world utilization. We propose a continual zero-shot learning model(A-CZSL) that is more suitable in real-case scenarios to address the issue that can learn sequentially and distinguish classes the model has not seen during training. Further, to enhance the reliability, we develop A-CZSL for a single hea…
University of Waterloo（滑铁卢大学）；Capital One（第一资本金融公司）
In this paper, we consider a challenging but realistic continual learning (CL) problem, Few-Shot Continual Active Learning (FoCAL), where a CL agent is provided with unlabeled data for a new or a previously learned task in each increment and the agent only has limited labeling budget available. Towards this, we build on the continual learning and active learning literature and develop a framework that can allow a CL agent to continually learn new object classes from a few labeled training examples. Our framework represents each object class using a uniform Gaussian mixture model (GMM) and uses…
MIT Media Lab（麻省理工学院媒体实验室）；Birla Institute of Technology（贝拉理工学院）
Self-supervised or weakly supervised models trained on large-scale datasets have shown sample-efficient transfer to diverse datasets in few-shot settings. We consider how upstream pretrained models can be leveraged for downstream few-shot, multilabel, and continual learning tasks. Our model CLIPPER (CLIP PERsonalized) uses image representations from CLIP, a large-scale image representation learning model trained using weak natural language supervision. We developed a technique, called Multi-label Weight Imprinting (MWI), for multi-label, continual, and few-shot learning, and CLIPPER uses MWI w…
Deep learning models suffer from catastrophic forgetting when trained in an incremental learning setting. In this work, we propose a novel approach to address the task incremental learning problem, which involves training a model on new tasks that arrive in an incremental manner. The task incremental learning problem becomes even more challenging when the test set contains classes that are not part of the train set, i.e., a task incremental generalized zero-shot learning problem. Our approach can be used in both the zero-shot and non zero-shot task incremental learning settings. Our proposed m…
Syracuse University（雪城大学）；Northeastern University（东北大学）；Colorado School of Mines（科罗拉多矿业学院）
Humans have a remarkable ability to quickly and effectively learn new concepts in a continuous manner without forgetting old knowledge. Though deep learning has made tremendous successes on various computer vision tasks, it faces challenges for achieving such human-level intelligence. In this paper, we define a new problem called continual few-shot learning, in which tasks arrive sequentially and each task is associated with a few training samples. We propose Continual Meta-Learner (CML) to solve this problem. CML integrates metric-based classification and a memory-based mechanism along with a…
CL-CLIP: CLIP-Based Continual Learning Framework with Cost-Volume Category Decoupling for Object Detection
National College for Excellent Engineers, Beihang University（卓越工程师学院，北京航空航天大学）；AI Research, Qihoo 360（360人工智能研究院，奇虎360）；School of Electronic Information Engineering, Beihang University（电子信息学院，北京航空航天大学）；School of Cyber Science and Technology, Beihang University（网络安全科学与技术学院，北京航空航天大学）；School of Computer Science and Engineering, Beihang University（计算机科学与工程学院，北京航空航天大学）；State Key Laboratory of Media Convergence and Communication, Communication University of China（媒体融合与传播国家重点实验室，中国传媒大学）；Institute of Information Technology, Mathematics and Mechanics, Lobachebsky University（信息技术、数学与力学学院，洛瓦茨基大学）；School of Artificial Intelligence, Beihang University（人工智能学院，北京航空航天大学）
Continual Alignment for SAM: Rethinking Foundation Models for Medical Image Segmentation in Continual Learning
Xi’an Jiaotong University（西安交通大学）；School of Information and Communications Engineering, Xi’an Jiaotong University（信息与通信工程学院）；School of Mathematics and Statistics, Xi’an Jiaotong University（数学与统计学学院）
Zero-shot learning (ZSL) has been shown to be a promising approach to generalizing a model to categories unseen during training by leveraging class attributes, but challenges still remain. Recently, methods using generative models to combat bias towards classes seen during training have pushed the state of the art of ZSL, but these generative models can be slow or computationally expensive to train. Additionally, while many previous ZSL methods assume a one-time adaptation to unseen classes, in reality, the world is always changing, necessitating a constant adjustment for deployed models. Mode…
ZeroSlide: Is Zero-Shot Classification Adequate for Lifelong Learning in Whole-Slide Image Analysis in the Era of Pathology Vision-Language Foundation Models?
Nara Institute of Science and Technology（奈良科学技术研究所）
Institute of Electronics, National Chiao Tung University（国立交通大学电子研究所）
Deep neural networks (DNNs) are known to perform well when deployed to test distributions that shares high similarity with the training distribution. Feeding DNNs with new data sequentially that were unseen in the training distribution has two major challenges -- fast adaptation to new tasks and catastrophic forgetting of old tasks. Such difficulties paved way for the on-going research on few-shot learning and continual learning. To tackle these problems, we introduce Attentive Independent Mechanisms (AIM). We incorporate the idea of learning using fast and slow weights in conjunction with the…
Indian Institute of Science（印度科学学院）；Indian Institute of Technology Madras（印度理工学院马德拉斯分校）
Zero-shot learning is a new paradigm to classify objects from classes that are not available at training time. Zero-shot learning (ZSL) methods have attracted considerable attention in recent years because of their ability to classify unseen/novel class examples. Most of the existing approaches on ZSL works when all the samples from seen classes are available to train the model, which does not suit real life. In this paper, we tackle this hindrance by developing a generative replay-based continual ZSL (GRCZSL). The proposed method endows traditional ZSL to learn from streaming data and acquire…
Adaptive Additive Parameter Updates of Vision Transformers for Few-Shot Continual Learning
Department of Intelligent Systems and Robotics, University of West Florida（智能系统与机器人系，西佛罗里达大学）；Florida Institute For Human and Machine Cognition (IHMC)（佛罗里达人类与机器认知研究所）；Center for Cybersecurity, University of West Florida（网络安全中心，西佛罗里达大学）
Università di Trento（特伦托大学）；Università di Modena e Reggio Emilia（摩德纳雷焦艾米利亚大学）
Future deep learning systems call for techniques that can deal with the evolving nature of temporal data and scarcity of annotations when new problems occur. As a step towards this goal, we present FUSION (Few-shot UnSupervIsed cONtinual learning), a learning strategy that enables a neural network to learn quickly and continually on streams of unlabelled data and unbalanced tasks. The objective is to maximise the knowledge extracted from the unlabelled data stream (unsupervised), favor the forward transfer of previously learnt tasks and features (continual) and exploit as much as possible the …
Few-Shot Continual Learning for 3D Brain MRI with Frozen Foundation Models
Independent Researcher, USA（美国独立研究者）；Indiana University at Bloomington, United States（印第安纳大学布卢明顿分校）；Independent Researcher, Taiwan（台湾独立研究者）；National Taiwan University, Taiwan（台湾国立大学）；Boise State University, United States（博伊西州立大学）；National Yang Ming Chiao Tung University, Taiwan（台湾阳明交通大学）
Unlocking Compositional Generalization in Continual Few-Shot Learning
Faculty of Information Technology, University of Science, Vietnam National University（信息科技学院，科学大学，越南国家大学）；Department of Computer Science, University of Warwick（计算机科学系，沃里克大学）
King Abdullah University of Science and Technology (KAUST)（阿卜杜拉国王科技大学）；Moscow Institute of Physics and Technology (MIPT)（莫斯科物理技术学院）
Normalization techniques have proved to be a crucial ingredient of successful training in a traditional supervised learning regime. However, in the zero-shot learning (ZSL) world, these ideas have received only marginal attention. This work studies normalization in ZSL scenario from both theoretical and practical perspectives. First, we give a theoretical explanation to two popular tricks used in zero-shot learning: normalize+scale and attributes normalization and show that they help training by preserving variance during a forward pass. Next, we demonstrate that they are insufficient to norma…
The goal of few-shot learning is to learn a model that can recognize novel classes based on one or few training data. It is challenging mainly due to two aspects: (1) it lacks good feature representation of novel classes; (2) a few of labeled data could not accurately represent the true data distribution and thus it's hard to learn a good decision function for classification. In this work, we use a sophisticated network architecture to learn better feature representation and focus on the second issue. A novel continual local replacement strategy is proposed to address the data deficiency probl…
Beijing Key Laboratory of Intelligent Information Technology（智能信息处理北京市重点实验室）；Guangdong Laboratory of Machine Perception and Intelligent Computing（广东省机器感知与智能计算实验室）；Zhejiang University（浙江大学）
Indian Institute of Technology Hyderabad（印度理工学院海得拉巴分校）
The ability to continuously learn and adapt itself to new tasks, without losing grasp of already acquired knowledge is a hallmark of biological learning systems, which current deep learning systems fall short of. In this work, we present a novel methodology for continual learning called MERLIN: Meta-Consolidation for Continual Learning. We assume that weights of a neural network $\boldsymbol ψ$, for solving task $\boldsymbol t$, come from a meta-distribution $p(\boldsymbol{ψ|t})$. This meta-distribution is learned and consolidated incrementally. We operate in the challenging online continual l…
This paper presents a constructive algorithm that achieves successful one-shot learning of hidden spike-patterns in a competitive detection task. It has previously been shown (Masquelier et al., 2008) that spike-timing-dependent plasticity (STDP) and lateral inhibition can result in neurons competitively tuned to repeating spike-patterns concealed in high rates of overall presynaptic activity. One-shot construction of neurons with synapse weights calculated as estimates of converged STDP outcomes results in immediate selective detection of hidden spike-patterns. The capability of continual lea…
Computer Vision Center（计算机视觉中心）；Universitat Autònoma de Barcelona（巴塞罗那自治大学）；Nankai University（南开大学）；University of Florence（佛罗伦萨大学）；Huawei Kirin Solution（华为麒麟解决方案部）
Lifelong object re-identification incrementally learns from a stream of re-identification tasks. The objective is to learn a representation that can be applied to all tasks and that generalizes to previously unseen re-identification tasks. The main challenge is that at inference time the representation must generalize to previously unseen identities. To address this problem, we apply continual meta metric learning to lifelong object re-identification. To prevent forgetting of previous tasks, we use knowledge distillation and explore the roles of positive and negative pairs. Based on our observ…
Few-shot class-incremental learning (FSCIL) aims to design machine learning algorithms that can continually learn new concepts from a few data points, without forgetting knowledge of old classes. The difficulty lies in that limited data from new classes not only lead to significant overfitting issues but also exacerbate the notorious catastrophic forgetting problems. Moreover, as training data come in sequence in FSCIL, the learned classifier can only provide discriminative information in individual sessions, while FSCIL requires all classes to be involved for evaluation. In this paper, we add…
Advancing Cross-domain Discriminability in Continual Learning of Vision-Language Models
Institute of Science Tokyo（东京科学研究院）；University of California Berkeley（加州大学伯克利分校）；Nanyang Technological University（南洋理工大学）；South China University of Technology（华南理工大学）；Greater Bay Area Institute for Innovation, Hunan University（湖南大学粤港澳大湾区创新研究院）
University of Trento（特伦托大学）；University of Modena and Reggio Emilia（摩德纳和雷焦艾米利亚大学）
In real-world applications, data do not reflect the ones commonly used for neural networks training, since they are usually few, unlabeled and can be available as a stream. Hence many existing deep learning solutions suffer from a limited range of applications, in particular in the case of online streaming data that evolve over time. To narrow this gap, in this work we introduce a novel and complex setting involving unsupervised meta-continual learning with unbalanced tasks. These tasks are built through a clustering procedure applied to a fitted embedding space. We exploit a meta-learning sch…
Indian Institute of Science（印度科学学院）；IIT Madras（印度理工学院马德拉斯分校）
Methods proposed in the literature for zero-shot learning (ZSL) are typically suitable for offline learning and cannot continually learn from sequential streaming data. The sequential data comes in the form of tasks during training. Recently, a few attempts have been made to handle this issue and develop continual ZSL (CZSL) methods. However, these CZSL methods require clear task-boundary information between the tasks during training, which is not practically possible. This paper proposes a task-free (i.e., task-agnostic) CZSL method, which does not require any task information during continua…
Zhejiang University（浙江大学）；Alibaba-Zhejiang University Joint Research Institute of Frontier Technologies（阿里巴巴-浙江大学前沿技术联合研究院）
Continual learning is a longstanding research topic due to its crucial role in tackling continually arriving tasks. Up to now, the study of continual learning in computer vision is mainly restricted to convolutional neural networks (CNNs). However, recently there is a tendency that the newly emerging vision transformers (ViTs) are gradually dominating the field of computer vision, which leaves CNN-based continual learning lagging behind as they can suffer from severe performance degradation if straightforwardly applied to ViTs. In this paper, we study ViT-backed continual learning to strive fo…
China University of Petroleum Beijing (CUP)（中国石油大学（北京））
Continual learning aims to rapidly and continually learn the current task from a sequence of tasks. Compared to other kinds of methods, the methods based on experience replay have shown great advantages to overcome catastrophic forgetting. One common limitation of this method is the data imbalance between the previous and current tasks, which would further aggravate forgetting. Moreover, how to effectively address the stability-plasticity dilemma in this setting is also an urgent problem to be solved. In this paper, we overcome these challenges by proposing a novel framework called Meta-learni…
Few-Shot Domain Incremental Learning via Continual Vision-Language Consolidation
School of Computer Science and IT, Adelaide University, Australia（阿德莱德大学计算机科学与信息技术学院，澳大利亚）；Faculty of Engineering and IT, University of Technology Sydney (UTS), Australia（悉尼大学工程与信息技术学院（UTS），澳大利亚）；School of Artificial Intelligence and Computer Science, Nantong University, China（南通大学人工智能与计算机科学学院，中国）；College of Computing and Data Science, Nanyang Technological University, Singapore（南洋理工大学计算与数据科学学院，新加坡）
