Sequential Targeting: an incremental learning approach for data imbalance in text classification
Korea University（高丽大学）；NAVER；Yonsei University（延世大学）；Naver Corp.（Naver公司）；TU Kaiserslautern（凯泽斯劳滕工业大学）；Korea Institute of Science and Technology Europe Forschungsgesellschaft mbH（韩国科学技术研究院欧洲研究所有限公司）
Joel Jang, Yoonjeon Kim, Kyoungho Choi, Sungho Suh
备注：
9 pages, 7 figures, submitted to the journal of Expert Systems with Applications
Classification tasks require a balanced distribution of data to ensure the learner to be trained to generalize over all classes. In real-world datasets, however, the number of instances vary substantially among classes. This typically leads to a learner that promotes bias towards the majority group due to its dominating property. Therefore, methods to handle imbalanced datasets are crucial for alleviating distributional skews and fully utilizing the under-represented data, especially in text classification. While addressing the imbalance in text data, most methods utilize sampling methods on the numerical representation of the data, which limits its efficiency on how effective the representation is. We propose a novel training method, Sequential Targeting(ST), independent of the effectiveness of the representation method, which enforces an incremental learning setting by splitting the data into mutually exclusive subsets and training the learner adaptively. To address problems that arise within incremental learning, we apply elastic weight consolidation. We demonstrate the effectiveness of our method through experiments on simulated benchmark datasets (IMDB) and data collected from NAVER.
Continual or lifelong learning has been a long-standing challenge in machine learning to date, especially in natural language processing (NLP). Although state-of-the-art language models such as BERT have ushered in a new era in this field due to their outstanding performance in multitask learning scenarios, they suffer from forgetting when being exposed to a continuous stream of data with shifting data distributions. In this paper, we introduce DRILL, a novel continual learning architecture for open-domain text classification. DRILL leverages a biologically inspired self-organizing neural arch…
Deep learning approaches are successful in a wide range of AI problems and in particular for visual recognition tasks. However, there are still open problems among which is the capacity to handle streams of visual information and the management of class imbalance in datasets. Existing research approaches these two problems separately while they co-occur in real world applications. Here, we study the problem of learning incrementally from imbalanced datasets. We focus on algorithms which have a constant deep model complexity and use a bounded memory to store exemplars of old classes across incr…
Resilient Class-Incremental Learning: on the Interplay of Drifting, Unlabelled and Imbalanced Data Streams
KIOS Research and Innovation Center of Excellence, University of Cyprus（希腊塞浦路斯大学KIOS研究中心）；Department of Electrical and Computer Engineering, University of Cyprus（塞浦路斯大学电气与计算机工程系）
The performance of classification algorithms with a massive and highly imbalanced data stream depends upon efficient balancing strategy. Some techniques of balancing strategy have been applied in the past with Batch data to resolve the class imbalance problem. This paper proposes a new incremental data balancing framework which can work with massive imbalanced data streams. In this paper, we choose Racing Algorithm as an automated data balancing technique which optimizes the balancing techniques. We applied Random Forest classification algorithm which can deal with the massive data stream. We …
Zhejiang University（浙江大学）；University of Illinois Urbana–Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of Science and Technology Beijing（北京科技大学）
Intelligent fault diagnosis has made extraordinary advancements currently. Nonetheless, few works tackle class-incremental learning for fault diagnosis under limited fault data, i.e., imbalanced and long-tailed fault diagnosis, which brings about various notable challenges. Initially, it is difficult to extract discriminative features from limited fault data. Moreover, a well-trained model must be retrained from scratch to classify the samples from new classes, thus causing a high computational burden and time consumption. Furthermore, the model may suffer from catastrophic forgetting when tra…
Class Incremental Fault Diagnosis under Limited Fault Data via Supervised Contrastive Knowledge Distillation
ZJU-UIUC Joint Institute at Zhejiang University（浙江大学-伊利诺伊大学厄巴纳-香槟分校联合研究所）；ZJU-UoE Institute at Zhejiang University（浙江大学-爱丁堡大学联合研究所）；College of Biomedical Engineering and Instrument Science at Zhejiang University（浙江大学生物医学工程与仪器科学学院）；College of Computer Science and Technology in Zhejiang University（浙江大学计算机科学与技术学院）
In natural language understanding (NLU) production systems, users' evolving needs necessitate the addition of new features over time, indexed by new symbols added to the meaning representation space. This requires additional training data and results in ever-growing datasets. We present the first systematic investigation of this incremental symbol learning scenario. Our analysis reveals a troubling quirk in building broad-coverage NLU systems: as the training dataset grows, performance on the new symbol often decreases if we do not accordingly increase its training data. This suggests that it …
Computer Vision Center, Universitat Autonoma de Barcelona（巴塞罗那自治大学计算机视觉中心）；Media Integration and Communication Center, University of Florence（佛罗伦萨大学媒体集成与传播中心）；Huawei Kirin Solution（华为麒麟解决方案）
Humans are capable of learning new tasks without forgetting previous ones, while neural networks fail due to catastrophic forgetting between new and previously-learned tasks. We consider a class-incremental setting which means that the task-ID is unknown at inference time. The imbalance between old and new classes typically results in a bias of the network towards the newest ones. This imbalance problem can either be addressed by storing exemplars from previous tasks, or by using image replay methods. However, the latter can only be applied to toy datasets since image generation for complex da…
Class-wise Balancing Data Replay for Federated Class-Incremental Learning
School of Software, Shandong University（山东大学软件学院）；College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算机与数据科学学院）；Department of Electrical and Computer Engineering, University of British Columbia（不列颠哥伦比亚大学电气与计算机工程系）；Vector Institute（向量研究所）
Careful Seeding for k-Medois Clustering with Incremental k-Means++ Initialization
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院）；School of Data Science, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）数据科学学院）
Federated learning aims to collaboratively train models without accessing their client's local private data. The data may be Non-IID for different clients and thus resulting in poor performance. Recently, personalized federated learning (PFL) has achieved great success in handling Non-IID data by enforcing regularization in local optimization or improving the model aggregation scheme on the server. However, most of the PFL approaches do not take into account the unfair competition issue caused by the imbalanced data distribution and lack of positive samples for some classes in each client. To …
Sharif University of Technology（谢里夫理工大学）；Institute For Research In Fundamental Sciences (IPM)（基础科学研究所（IPM））
This paper presents a practical and simple yet efficient method to effectively deal with the catastrophic forgetting for Class Incremental Learning (CIL) tasks. CIL tends to learn new concepts perfectly, but not at the expense of performance and accuracy for old data. Learning new knowledge in the absence of data instances from previous classes or even imbalance samples of both old and new classes makes CIL an ongoing challenging problem. These issues can be tackled by storing exemplars belonging to the previous tasks or by utilizing the rehearsal strategy. Inspired by the rehearsal strategy w…
Adaptive Adapter Routing for Long-Tailed Class-Incremental Learning
School of Artificial Intelligence, Nanjing University（南京大学人工智能学院）；National Key Laboratory for Novel Software Technology, Nanjing University（南京大学计算机软件新技术国家重点实验室）
Neural Collapse Terminus: A Unified Solution for Class Incremental Learning and Its Variants
King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）；Wuhan University（武汉大学）；Peking University（北京大学）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））；University of Oxford（牛津大学）；University of Sydney（悉尼大学）
Shenyang Institute of Automation, Chinese Academy of Sciences（中国科学院沈阳自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Northwestern University（西北大学）；University of Technology Sydney（悉尼科技大学）
Federated learning (FL) has attracted growing attention via data-private collaborative training on decentralized clients. However, most existing methods unrealistically assume object classes of the overall framework are fixed over time. It makes the global model suffer from significant catastrophic forgetting on old classes in real-world scenarios, where local clients often collect new classes continuously and have very limited storage memory to store old classes. Moreover, new clients with unseen new classes may participate in the FL training, further aggravating the catastrophic forgetting o…
Gwangju Institute of Science and Technology (GIST)（光州科学技术院）
Incremental learning suffers from two challenging problems; forgetting of old knowledge and intransigence on learning new knowledge. Prediction by the model incrementally learned with a subset of the dataset are thus uncertain and the uncertainty accumulates through the tasks by knowledge transfer. To prevent overfitting to the uncertain knowledge, we propose to penalize confident fitting to the uncertain knowledge by the Maximum Entropy Regularizer (MER). Additionally, to reduce class imbalance and induce a self-paced curriculum on new classes, we exclude a few samples from the new classes in…
L3S Research Center（L3S研究中心）；Leibniz University of Hanover（汉诺威莱布尼茨大学）
Online class imbalance learning deals with data streams that are affected by both concept drift and class imbalance. Online learning tries to find a trade-off between exploiting previously learned information and incorporating new information into the model. This requires both the incremental update of the model and the ability to unlearn outdated information. The improper use of unlearning, however, can lead to the retroactive interference problem, a phenomenon that occurs when newly learned information interferes with the old information and impedes the recall of previously learned informati…
This paper considers incremental few-shot learning, which requires a model to continually recognize new categories with only a few examples provided. Our study shows that existing methods severely suffer from catastrophic forgetting, a well-known problem in incremental learning, which is aggravated due to data scarcity and imbalance in the few-shot setting. Our analysis further suggests that to prevent catastrophic forgetting, actions need to be taken in the primitive stage -- the training of base classes instead of later few-shot learning sessions. Therefore, we propose to search for flat loc…
Tokyo Institute of Technology（东京工业大学）；Artificial Intelligence Research Center, AIST（产业技术综合研究所人工智能研究中心）
When incrementally trained on new classes, deep neural networks are subject to catastrophic forgetting which leads to an extreme deterioration of their performance on the old classes while learning the new ones. Using a small memory containing few samples from past classes has shown to be an effective method to mitigate catastrophic forgetting. However, due to the limited size of the replay memory, there is a large imbalance between the number of samples for the new and the old classes in the training dataset resulting in bias in the final model. To address this issue, we propose to use the Ba…
3D semantic scene labeling is a fundamental task for Autonomous Driving. Recent work shows the capability of Deep Neural Networks in labeling 3D point sets provided by sensors like LiDAR, and Radar. Imbalanced distribution of classes in the dataset is one of the challenges that face 3D semantic scene labeling task. This leads to misclassifying for the non-dominant classes which suffer from two main problems: a) rare appearance in the dataset, and b) few sensor points reflected from one object of these classes. This paper proposes a Weighted Self-Incremental Transfer Learning as a generalized m…
The University of British Columbia（不列颠哥伦比亚大学）；Vancouver General Hospital（温哥华综合医院）
Standard deep learning-based classification approaches require collecting all samples from all classes in advance and are trained offline. This paradigm may not be practical in real-world clinical applications, where new classes are incrementally introduced through the addition of new data. Class incremental learning is a strategy allowing learning from such data. However, a major challenge is catastrophic forgetting, i.e., performance degradation on previous classes when adapting a trained model to new data. Prior methodologies to alleviate this challenge save a portion of training data requi…
Training on class-imbalanced data usually results in biased models that tend to predict samples into the majority classes, which is a common and notorious problem. From the perspective of energy-based model, we demonstrate that the free energies of categories are aligned with the label distribution theoretically, thus the energies of different classes are expected to be close to each other when aiming for ``balanced'' performance. However, we discover a severe energy-bias phenomenon in the models trained on class-imbalanced dataset. To eliminate the bias, we propose a simple and effective meth…
Incremental Outlier Detection Modelling Using Streaming Analytics in Finance & Health Care
National Institute of Technology Warangal（瓦朗加尔国家理工学院）；Jawaharlal Nehru Technological University – Gurajada Vizianagaram (JNTU-GV)（贾瓦哈拉尔·尼赫鲁技术大学古拉贾达维齐亚纳格拉姆分校）
Cisco Systems, Inc.（思科系统公司）；Czech Technical University in Prague（布拉格捷克理工大学）；Charles University（查理大学）
Severe class imbalance is one of the main conditions that make machine learning in cybersecurity difficult. A variety of dataset preprocessing methods have been introduced over the years. These methods modify the training dataset by oversampling, undersampling or a combination of both to improve the predictive performance of classifiers trained on this dataset. Although these methods are used in cybersecurity occasionally, a comprehensive, unbiased benchmark comparing their performance over a variety of cybersecurity problems is missing. This paper presents a benchmark of 16 preprocessing meth…
The Hong Kong Polytechnic University（香港理工大学）；Huazhong University of Science and Technology（华中科技大学）；Beijing Institute of Technology（北京理工大学）；KAUST（阿卜杜拉国王科技大学）；SDAIA-KAUST AI Center（沙特数据与人工智能局-阿卜杜拉国王科技大学人工智能中心）
Online Class-Incremental (OCI) learning has sparked new approaches to expand the previously trained model knowledge from sequentially arriving data streams with new classes. Unfortunately, OCI learning can suffer from catastrophic forgetting (CF) as the decision boundaries for old classes can become inaccurate when perturbated by new ones. Existing literature have applied the data augmentation (DA) to alleviate the model forgetting, while the role of DA in OCI has not been well understood so far. In this paper, we theoretically show that augmented samples with lower correlation to the original…
A Hybrid Active-Passive Approach to Imbalanced Nonstationary Data Stream Classification
KIOS Research and Innovation Center of Excellence（KIOS研究与创新中心）；Department of Electrical and Computer Engineering（电气与计算机工程系）；University of Cyprus（塞浦路斯大学）
Incremental Causal Graph Learning for Online Cyberattack Detection in Cyber-Physical Infrastructures
School of Computing and Augmented Intelligence, Arizona State University（计算与增强智能学院，亚利桑那州立大学）；Institute for Information Sciences, University of Kansas（信息科学研究所，堪萨斯大学）
University of Texas at Austin（德克萨斯大学奥斯汀分校）；Michigan State University（密歇根州立大学）；MIT-IBM Watson AI Lab, IBM Research（IBM研究MIT-IBM沃森人工智能实验室）；University of California, Santa Barbara（加州大学圣巴巴拉分校）
Class-incremental learning (CIL) suffers from the notorious dilemma between learning newly added classes and preserving previously learned class knowledge. That catastrophic forgetting issue could be mitigated by storing historical data for replay, which yet would cause memory overheads as well as imbalanced prediction updates. To address this dilemma, we propose to leverage "free" external unlabeled data querying in continual learning. We first present a CIL with Queried Unlabeled Data (CIL-QUD) scheme, where we only store a handful of past training samples as anchors and use them to query re…
Few-shot Class-incremental Fault Diagnosis by Preserving Class-Agnostic Knowledge with Dual-Granularity Representations
School of Computer and Software Engineering, Xihua University（西华大学计算机与软件工程学院）；School of Computing and Artificial Intelligence, Southwest Jiaotong University（西南交通大学计算机与人工智能学院）；School of Computer Science and Artificial Intelligence, Southwest Petroleum University（西南石油大学计算机科学与人工智能学院）；School of Automation Engineering, University of Electronic Science and Technology of China（电子科技大学自动化工程学院）；State Key Laboratory of Rail Transit Vehicle System（轨道交通车辆系统国家重点实验室）
Amirkabir University of Technology (Tehran Polytechnic)（阿米尔卡比尔理工大学（德黑兰理工大学））
Credit scoring models, which are among the most potent risk management tools that banks and financial institutes rely on, have been a popular subject for research in the past few decades. Accordingly, many approaches have been developed to address the challenges in classifying loan applicants and improve and facilitate decision-making. The imbalanced nature of credit scoring datasets, as well as the heterogeneous nature of features in credit scoring datasets, pose difficulties in developing and implementing effective credit scoring models, targeting the generalization power of classification m…
In this era of digital information explosion, an abundance of data from numerous modalities is being generated as well as archived everyday. However, most problems associated with training Deep Neural Networks still revolve around lack of data that is rich enough for a given task. Data is required not only for training an initial model, but also for future learning tasks such as Model Compression and Incremental Learning. A diverse dataset may be used for training an initial model, but it may not be feasible to store it throughout the product life cycle due to data privacy issues or memory con…
