Incremental Learning (IL) is useful when artificial systems need to deal with streams of data and do not have access to all data at all times. The most challenging setting requires a constant complexity of the deep model and an incremental model update without access to a bounded memory of past data. Then, the representations of past classes are strongly affected by catastrophic forgetting. To mitigate its negative effect, an adapted fine tuning which includes knowledge distillation is usually deployed. We propose a different approach based on a vanilla fine tuning backbone. It leverages initial classifier weights which provide a strong representation of past classes because they are trained with all class data. However, the magnitude of classifiers learned in different states varies and normalization is needed for a fair handling of all classes. Normalization is performed by standardizing the initial classifier weights, which are assumed to be normally distributed. In addition, a calibration of prediction scores is done by using state level statistics to further improve classification fairness. We conduct a thorough evaluation with four public datasets in a memoryless incremental learning setting. Results show that our method outperforms existing techniques by a large margin for large-scale datasets.
Noah’s Ark Lab, Huawei Technologies（华为技术诺亚方舟实验室）；McMaster University（麦克马斯特大学）；University of Manitoba（曼尼托巴大学）
Few-shot class-incremental learning (FSCIL) has been proposed aiming to enable a deep learning system to incrementally learn new classes with limited data. Recently, a pioneer claims that the commonly used replay-based method in class-incremental learning (CIL) is ineffective and thus not preferred for FSCIL. This has, if truth, a significant influence on the fields of FSCIL. In this paper, we show through empirical results that adopting the data replay is surprisingly favorable. However, storing and replaying old data can lead to a privacy concern. To address this issue, we alternatively prop…
College of Computer Science, Nankai University（南开大学计算机学院）；MICC, University of Florence（佛罗伦萨大学MICC）；Huawei Kirin Solution（华为麒麟解决方案）
Most meta-learning approaches assume the existence of a very large set of labeled data available for episodic meta-learning of base knowledge. This contrasts with the more realistic continual learning paradigm in which data arrives incrementally in the form of tasks containing disjoint classes. In this paper we consider this problem of Incremental Meta-Learning (IML) in which classes are presented incrementally in discrete tasks. We propose an approach to IML, which we call Episodic Replay Distillation (ERD), that mixes classes from the current task with class exemplars from previous tasks whe…
Indian Institute of Technology Bombay（印度理工学院孟买分校）；Goethe University Frankfurt（法兰克福歌德大学）
We tackle the novel problem of incremental unsupervised domain adaptation (IDA) in this paper. We assume that a labeled source domain and different unlabeled target domains are incrementally observed with the constraint that data corresponding to the current domain is only available at a time. The goal is to preserve the accuracies for all the past domains while generalizing well for the current domain. The IDA setup suffers due to the abrupt differences among the domains and the unavailability of past data including the source domain. Inspired by the notion of generative feature replay, we pr…
This paper proposes two novel knowledge transfer techniques for class-incremental learning (CIL). First, we propose data-free generative replay (DF-GR) to mitigate catastrophic forgetting in CIL by using synthetic samples from a generative model. In the conventional generative replay, the generative model is pre-trained for old data and shared in extra memory for later incremental learning. In our proposed DF-GR, we train a generative model from scratch without using any training data, based on the pre-trained classification model from the past, so we curtail the cost of sharing pre-trained ge…
IBM Research Europe（IBM欧洲研究院）；ETH Zurich（苏黎世联邦理工学院）；Uppsala University（乌普萨拉大学）
Neural networks are prone to catastrophic forgetting when trained incrementally on different tasks. Popular incremental learning methods mitigate such forgetting by retaining a subset of previously seen samples and replaying them during the training on subsequent tasks. However, this is not always possible, e.g., due to data protection regulations. In such restricted scenarios, one can employ generative models to replay either artificial images or hidden features to a classifier. In this work, we propose Genifer (GENeratIve FEature-driven image Replay), where a generative model is trained to r…
Computer Vision Center, Universitat Autonoma de Barcelona（巴塞罗那自治大学计算机视觉中心）；Media Integration and Communication Center, University of Florence（佛罗伦萨大学媒体集成与传播中心）；Huawei Kirin Solution（华为麒麟解决方案）
Humans are capable of learning new tasks without forgetting previous ones, while neural networks fail due to catastrophic forgetting between new and previously-learned tasks. We consider a class-incremental setting which means that the task-ID is unknown at inference time. The imbalance between old and new classes typically results in a bias of the network towards the newest ones. This imbalance problem can either be addressed by storing exemplars from previous tasks, or by using image replay methods. However, the latter can only be applied to toy datasets since image generation for complex da…
University of Toronto（多伦多大学）；LG AI Research（LG人工智能研究院）
Online class-incremental continual learning (CL) studies the problem of learning new classes continually from an online non-stationary data stream, intending to adapt to new data while mitigating catastrophic forgetting. While memory replay has shown promising results, the recency bias in online learning caused by the commonly used Softmax classifier remains an unsolved challenge. Although the Nearest-Class-Mean (NCM) classifier is significantly undervalued in the CL community, we demonstrate that it is a simple yet effective substitute for the Softmax classifier. It addresses the recency bias…
Modern machine learning systems need to be able to cope with constantly arriving and changing data. Two main areas of research dealing with such scenarios are continual learning and data stream mining. Continual learning focuses on accumulating knowledge and avoiding forgetting, assuming information once learned should be stored. Data stream mining focuses on adaptation to concept drift and discarding outdated information, assuming that only the most recent data is relevant. While these two areas are mainly being developed in separation, they offer complementary views on the problem of learnin…
ShanghaiTech University（上海科技大学）；Shanghai Institute of Microsystem and Information Technology, Chinese Academy of Sciences（中国科学院上海微系统与信息技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）；KU Leuven（鲁汶大学）；Shanghai Engineering Research Center of Intelligent Vision and Imaging（上海智能视觉成像工程技术研究中心）
Vision-language pre-training (VLP) has attracted increasing attention recently. With a large amount of image-text pairs, VLP models trained with contrastive loss have achieved impressive performance in various tasks, especially the zero-shot generalization on downstream datasets. In practical applications, however, massive data are usually collected in a streaming fashion, requiring VLP models to continuously integrate novel knowledge from incoming data and retain learned knowledge. In this work, we focus on learning a VLP model with sequential chunks of image-text pair data. To tackle the cat…
Porsche Engineering Services GmbH（保时捷工程服务有限公司）；Fraunhofer IOSB（弗劳恩霍夫光学、系统技术与图像利用研究所）；Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）
The field of continual deep learning is an emerging field and a lot of progress has been made. However, concurrently most of the approaches are only tested on the task of image classification, which is not relevant in the field of intelligent vehicles. Only recently approaches for class-incremental semantic segmentation were proposed. However, all of those approaches are based on some form of knowledge distillation. At the moment there are no investigations on replay-based approaches that are commonly used for object recognition in a continual setting. At the same time while unsupervised domai…
Adaptive Visual Scene Understanding: Incremental Scene Graph Generation
College of Computing and Data Science, Nanyang Technological University (NTU)（南洋理工大学计算与数据科学学院）；Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局）
University of Bologna（博洛尼亚大学）；University of Pisa（比萨大学）
Learning continually is a key aspect of intelligence and a necessary ability to solve many real-life problems. One of the most effective strategies to control catastrophic forgetting, the Achilles' heel of continual learning, is storing part of the old data and replaying them interleaved with new experiences (also known as the replay approach). Generative replay, which is using generative models to provide replay patterns on demand, is particularly intriguing, however, it was shown to be effective mainly under simplified assumptions, such as simple scenarios and low-dimensional data. In this p…
University of Toronto（多伦多大学）；LG AI Research（LG人工智能研究院）
As image-based deep learning becomes pervasive on every device, from cell phones to smart watches, there is a growing need to develop methods that continually learn from data while minimizing memory footprint and power consumption. While memory replay techniques have shown exceptional promise for this task of continual learning, the best method for selecting which buffered images to replay is still an open question. In this paper, we specifically focus on the online class-incremental setting where a model needs to learn new classes continually from an online data stream. To this end, we contri…
IIT Kanpur（印度理工学院坎普尔分校）；Duke University（杜克大学）；Verisk Analytics（威斯科分析公司）；University of Bath（巴斯大学）
A wide variety of methods have been developed to enable lifelong learning in conventional deep neural networks. However, to succeed, these methods require a `batch' of samples to be available and visited multiple times during training. While this works well in a static setting, these methods continue to suffer in a more realistic situation where data arrives in \emph{online streaming manner}. We empirically demonstrate that the performance of current approaches degrades if the input is obtained as a stream of data with the following restrictions: $(i)$ each instance comes one at a time and can…
Attention-based Generative Latent Replay: A Continual Learning Approach for WSI Analysis
Faculty of Informatics and Data Science, University of Regensburg（信息学院与数据科学学院，雷根施塔特大学）；Institute of Pathology, Hannover Medical School（病理学研究所，汉诺威医学院）；Fraunhofer Institute for Digital Medicine MEVIS（弗劳恩霍夫数字医学研究所MEVIS）
针对 WSI 分类中的领域漂移问题，提出 AGLR-CL 框架，利用 GMM 合成表示并结合注意力过滤实现无缓冲、隐私保护的持续学习。
Indian Institute of Science（印度科学学院）；Indian Institute of Technology Madras（印度理工学院马德拉斯分校）
Zero-shot learning is a new paradigm to classify objects from classes that are not available at training time. Zero-shot learning (ZSL) methods have attracted considerable attention in recent years because of their ability to classify unseen/novel class examples. Most of the existing approaches on ZSL works when all the samples from seen classes are available to train the model, which does not suit real life. In this paper, we tackle this hindrance by developing a generative replay-based continual ZSL (GRCZSL). The proposed method endows traditional ZSL to learn from streaming data and acquire…
Computer Aided Medical Procedures, Technische Universität München（慕尼黑工业大学计算机辅助医疗程序）；Canon Medical Research Europe ltd.（佳能医疗研究欧洲有限公司）；Computer Aided Medical Procedures, Johns Hopkins University, Baltimore, USA（约翰斯·霍普金斯大学计算机辅助医疗程序）
Deep learning organ segmentation approaches require large amounts of annotated training data, which is limited in supply due to reasons of confidentiality and the time required for expert manual annotation. Therefore, being able to train models incrementally without having access to previously used data is desirable. A common form of sequential training is fine tuning (FT). In this setting, a model learns a new task effectively, but loses performance on previously learned tasks. The Learning without Forgetting (LwF) approach addresses this issue via replaying its own prediction for past tasks …
Deep learning has achieved remarkable success in object recognition tasks through the availability of large scale datasets like ImageNet. However, deep learning systems suffer from catastrophic forgetting when learning incrementally without replaying old data. For real-world applications, robots also need to incrementally learn new objects. Further, since robots have limited human assistance available, they must learn from only a few examples. However, very few object recognition datasets and benchmarks exist to test incremental learning capability for robotic vision. Further, there is no data…
This paper studies class incremental learning (CIL) of continual learning (CL). Many approaches have been proposed to deal with catastrophic forgetting (CF) in CIL. Most methods incrementally construct a single classifier for all classes of all tasks in a single head network. To prevent CF, a popular approach is to memorize a small number of samples from previous tasks and replay them during training of the new task. However, this approach still suffers from serious CF as the parameters learned for previous tasks are updated or adjusted with only the limited number of saved samples in the memo…
IBM Research Europe（IBM欧洲研究院）；ETH Zurich（苏黎世联邦理工学院）；Uppsala University（乌普萨拉大学）
Deep learning models can achieve high accuracy when trained on large amounts of labeled data. However, real-world scenarios often involve several challenges: Training data may become available in installments, may originate from multiple different domains, and may not contain labels for training. Certain settings, for instance medical applications, often involve further restrictions that prohibit retention of previously seen data due to privacy regulations. In this work, to address such challenges, we study unsupervised segmentation in continual learning scenarios that involve domain shift. To…
InfoDense: Density-Aware Regional Decisive Replay for Memory-Efficient Incremental Face Forgery Detection
Huazhong Agricultural University（华中农业大学）；Peking University（北京大学）；National University of Singapore（新加坡国立大学）；Nantong University（南通大学）；Wuhan University（武汉大学）
FETCH: A Memory-Efficient Replay Approach for Continual Learning in Image Classification
Faculty of Electrical Engineering and Information Technology, Chemnitz University of Technology（德国 Chemnitz 工业大学 电气工程与信息技术学院）；Institute for Computational Visualistics, University of Koblenz（科布伦茨大学 计算视觉学研究所）
Continual All-in-One Adverse Weather Removal with Knowledge Replay on a Unified Network Structure
School of Telecommunications Engineering, Xidian University（西安电子科技大学通信工程学院）；School of Automation, Northwestern Polytechnical University（西北工业大学自动化学院）；Hefei Comprehensive National Science Center（合肥综合性国家科学中心）；University of New South Wales (UNSW)（新南威尔士大学）；Shandong Normal University（山东师范大学）
Domain-incremental White Blood Cell Classification with Privacy-aware Continual Learning
University of Regensburg（雷根斯堡大学）；University Hospital RWTH Aachen（亚琛工业大学附属医院）；University Hospital Erlangen（埃尔朗根大学附属医院）；Fraunhofer Institute for Digital Medicine MEVIS（弗劳恩霍夫数字医学梅维斯研究所）
Artificial Intelligence, Analytics And Informatics (AI), A*STAR（A*STAR人工智能、分析与信息学）；RMIT University（皇家墨尔本理工大学）；Institute for Infocomm Research (IR), A*STAR（A*STAR资讯通信研究院）
Deep neural networks have shown promise in several domains, and the learned data (task) specific information is implicitly stored in the network parameters. Extraction and utilization of encoded knowledge representations are vital when data is no longer available in the future, especially in a continual learning scenario. In this work, we introduce {\em flashcards}, which are visual representations that {\em capture} the encoded knowledge of a network as a recursive function of predefined random image patterns. In a continual learning scenario, flashcards help to prevent catastrophic forgettin…
University of Pisa（比萨大学）；University of Bologna（博洛尼亚大学）
In recent years we have witnessed a renewed interest in machine learning methodologies, especially for deep representation learning, that could overcome basic i.i.d. assumptions and tackle non-stationary environments subject to various distributional shifts or sample selection biases. Within this context, several computational approaches based on architectural priors, regularizers and replay policies have been proposed with different degrees of success depending on the specific scenario in which they were developed and assessed. However, designing comprehensive hybrid solutions that can flexib…
Learning on the Fly: Replay-Based Continual Object Perception for Indoor Drones
National University of Science and Technology Politehnica Bucharest, Romania（罗马尼亚布加勒斯特多项技术大学）；Institute of Mathematics ``Simion Stoilow'' of the Romanian Academy, Romania（罗马尼亚科学院模拟昂斯数学研究所）；NORCE Norwegian Research Center, Norway（挪威NORCE研究中心）
The Chinese University of Hong Kong（香港中文大学）；The University of Hong Kong（香港大学）
Contemporary methods have shown promising results on cardiac image segmentation, but merely in static learning, i.e., optimizing the network once for all, ignoring potential needs for model updating. In real-world scenarios, new data continues to be gathered from multiple institutions over time and new demands keep growing to pursue more satisfying performance. The desired model should incrementally learn from each incoming dataset and progressively update with improved functionality as time goes by. As the datasets sequentially delivered from multiple sites are normally heterogenous with doma…
Porsche Engineering Group GmbH（保时捷工程集团有限公司）；Fraunhofer Institute of Optronics, Systems Technologies and Image Exploitation IOSB（弗劳恩霍夫光电、系统技术与图像利用研究所IOSB）；Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）
Continual learning for Semantic Segmentation (CSS) is a rapidly emerging field, in which the capabilities of the segmentation model are incrementally improved by learning new classes or new domains. A central challenge in Continual Learning is overcoming the effects of catastrophic forgetting, which refers to the sudden drop in accuracy on previously learned tasks after the model is trained on new classes or domains. In continual classification this challenge is often overcome by replaying a small selection of samples from previous tasks, however replay is rarely considered in CSS. Therefore, …
Interdisciplinary Program in Artificial Intelligence, Seoul National University（人工智能交叉学科项目，首尔国立大学）；AI Institute for Seoul National University（首尔国立大学人工智能研究所）；SK Telecom（SK电信）；Korea Institute of Science and Technology（韩国科学技术院）
Training deep neural networks at the edge on light computational devices, embedded systems and robotic platforms is nowadays very challenging. Continual learning techniques, where complex models are incrementally trained on small batches of new data, can make the learning problem tractable even for CPU-only embedded devices enabling remarkable levels of adaptiveness and autonomy. However, a number of practical problems need to be solved: catastrophic forgetting before anything else. In this paper we introduce an original technique named "Latent Replay" where, instead of storing a portion of pa…
DiffusePast: Diffusion-based Generative Replay for Class Incremental Semantic Segmentation
The Hong Kong Polytechnic University（香港理工大学）；Center for Artificial Intelligence and Robotics, HKISI, CAS（中国科学院香港创新研究院人工智能与机器人创新中心）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）
