The $F_β$ score is a commonly used measure of classification performance, which plays crucial roles in classification tasks with imbalanced data sets. However, the $F_β$ score cannot be used as a loss function by gradient-based learning algorithms for optimizing neural network parameters due to its non-differentiability. On the other hand, commonly used loss functions such as the binary cross-entropy (BCE) loss are not directly related to performance measures such as the $F_β$ score, so that neural networks optimized by using the loss functions may not yield optimal performance measures. In this study, we investigate a relationship between classification performance measures and loss functions in terms of the gradients with respect to the model parameters. Then, we propose a differentiable surrogate loss function for the optimization of the $F_β$ score. We show that the gradient paths of the proposed surrogate $F_β$ loss function approximate the gradient paths of the large sample limit of the $F_β$ score. Through numerical experiments using ResNets and benchmark image data sets, it is demonstrated that the proposed surrogate $F_β$ loss function is effective for optimizing $F_β$ scores under class imbalances in binary classification tasks compared with other loss functions.
University of Rostock（罗斯托克大学）；Grenoble Alps University（格勒诺布尔阿尔卑斯大学）；Grenoble Institute of Technology（格勒诺布尔理工学院）；Stellenbosch Institute for Advanced Study (STIAS)（斯泰伦博斯高等研究院）；Stellenbosch University（斯泰伦博斯大学）
The Synthetic Minority Oversampling TEchnique (SMOTE) is widely-used for the analysis of imbalanced datasets. It is known that SMOTE frequently over-generalizes the minority class, leading to misclassifications for the majority class, and effecting the overall balance of the model. In this article, we present an approach that overcomes this limitation of SMOTE, employing Localized Random Affine Shadowsampling (LoRAS) to oversample from an approximated data manifold of the minority class. We benchmarked our algorithm with 14 publicly available imbalanced datasets using three different Machine L…
Imbalanced datasets are ubiquitous. Classification performance on imbalanced datasets is generally poor for the minority class as the classifier cannot learn decision boundaries well. However, in sensitive applications like fraud detection, medical diagnosis, and spam identification, it is extremely important to classify the minority instances correctly. In this paper, we present a novel technique based on genetic algorithms, GenSample, for oversampling the minority class in imbalanced datasets. GenSample decides the rate of oversampling a minority example by taking into account the difficulty…
Hong Kong Baptist University（香港浸会大学）；University of Macau（澳门大学）
Recent studies have shown that imbalance ratio is not the only cause of the performance loss of a classifier in imbalanced data classification. In fact, other data factors, such as small disjuncts, noises and overlapping, also play the roles in tandem with imbalance ratio, which makes the problem difficult. Thus far, the empirical studies have demonstrated the relationship between the imbalance ratio and other data factors only. To the best of our knowledge, there is no any measurement about the extent of influence of class imbalance on the classification performance of imbalanced data. Furthe…
Kernel-Based Enhanced Oversampling Method for Imbalanced Classification
Faculty of Science and Technology, BNU-HKBU United International College（北师港浸会大学科技与文学院）；Guangdong Provincial Key Laboratory of Interdisciplinary Research and Application for Data Science, BNU-HKBU United International College（广东省数据科学跨学科研究与应用重点实验室（北师港浸会大学））
Tsinghua University（清华大学）；Institute for Interdisciplinary Information Sciences（交叉信息研究院）
Conductor galloping is the high-amplitude, low-frequency oscillation of overhead power lines due to wind. Such movements may lead to severe damages to transmission lines, and hence pose significant risks to the power system operation. In this paper, we target to design a prediction framework for conductor galloping. The difficulty comes from imbalanced dataset as galloping happens rarely. By examining the impacts of data balance and data volume on the prediction performance, we propose to employ proper sample adjustment methods to achieve better performance. Numerical study suggests that using…
University of Granada（格拉纳达大学）；University of Jaén（哈恩大学）
Multilabel classification is an emergent data mining task with a broad range of real world applications. Learning from imbalanced multilabel data is being deeply studied latterly, and several resampling methods have been proposed in the literature. The unequal label distribution in most multilabel datasets, with disparate imbalance levels, could be a handicap while learning new classifiers. In addition, this characteristic challenges many of the existent preprocessing algorithms. Furthermore, the concurrence between imbalanced labels can make harder the learning from certain labels. These are …
University of Birmingham（伯明翰大学）；University of Warwick（华威大学）；Alan Turing Institute（阿兰·图灵研究所）
The automatic identification of propaganda has gained significance in recent years due to technological and social changes in the way news is generated and consumed. That this task can be addressed effectively using BERT, a powerful new architecture which can be fine-tuned for text classification tasks, is not surprising. However, propaganda detection, like other tasks that deal with news documents and other forms of decontextualized social communication (e.g. sentiment analysis), inherently deals with data whose categories are simultaneously imbalanced and dissimilar. We show that BERT, while…
University of Rostock（罗斯托克大学）；Leibniz-Institute for Food Systems Biology（莱布尼茨食品系统生物学研究所）；Technical University of Munich（慕尼黑工业大学）；Royal Holloway, University of London（伦敦大学皇家霍洛威学院）
Over 85 oversampling algorithms, mostly extensions of the SMOTE algorithm, have been built over the past two decades, to solve the problem of imbalanced datasets. However, it has been evident from previous studies that different oversampling algorithms have different degrees of efficiency with different classifiers. With numerous algorithms available, it is difficult to decide on an oversampling algorithm for a chosen classifier. Here, we overcome this problem with a multi-schematic and classifier-independent oversampling approach: ProWRAS(Proximity Weighted Random Affine Shadowsampling). ProW…
American International University-Bangladesh（孟加拉国美国国际大学）
Imbalanced datasets can lead to biasedness into the detection of fake news. In this work, we present several strategies for resolving the imbalance issue for fake news detection in Bangla with a comparative assessment of proposed methodologies. Additionally, we propose a technique for improving performance even when the dataset is imbalanced. We applied our proposed approaches to BanFakeNews, a dataset developed for the purpose of detecting fake news in Bangla comprising of 50K instances but is significantly skewed, with 97% of majority instances. We obtained a 93.1% F1-score using data manipu…
Imperial College London（帝国理工学院）；University of Oxford（牛津大学）；University of Birmingham（伯明翰大学）
Machine learning models are typically deployed in a test setting that differs from the training setting, potentially leading to decreased model performance because of domain shift. If we could estimate the performance that a pre-trained model would achieve on data from a specific deployment setting, for example a certain clinic, we could judge whether the model could safely be deployed or if its performance degrades unacceptably on the specific data. Existing approaches estimate this based on the confidence of predictions made on unlabeled test data from the deployment's domain. We find existi…
Argonne National Laboratory（阿贡国家实验室）；Brookhaven National Laboratory（布鲁克海文国家实验室）
We propose a benchmark to study surrogate model accuracy for protein-ligand docking. We share a dataset consisting of 200 million 3D complex structures and 2D structure scores across a consistent set of 13 million "in-stock" molecules over 15 receptors, or binding sites, across the SARS-CoV-2 proteome. Our work shows surrogate docking models have six orders of magnitude more throughput than standard docking protocols on the same supercomputer node types. We demonstrate the power of high-speed surrogate models by running each target against 1 billion molecules in under a day (50k predictions pe…
When Does Synthetic Data Augmentation Improve Score-Based Imbalanced Classification?
Department of Electrical & Computer Engineering, Duke University（电气与计算机工程系，杜克大学）；Department of Biostatistics & Bioinformatics, Duke University（生物统计学与生物信息学系，杜克大学）；Department of Biostatistics & Bioinformatics and Department of Computer Science, Duke University（生物统计学与生物信息学系和计算机科学系，杜克大学）
Classification of imbalanced data is one of the common problems in the recent field of data mining. Imbalanced data substantially affects the performance of standard classification models. Data-level approaches mainly use the oversampling methods to solve the problem, such as synthetic minority oversampling Technique (SMOTE). However, since the methods such as SMOTE generate instances by linear interpolation, synthetic data space may look like a polygonal. Also, the oversampling methods generate outliers of the minority class. In this paper, we proposed Gaussian based minority oversampling tec…
The University of Tokyo（东京大学）；RIKEN AIP（理化学研究所先进智能项目中心）
Complex classification performance metrics such as the F${}_β$-measure and Jaccard index are often used, in order to handle class-imbalanced cases such as information retrieval and image segmentation. These performance metrics are not decomposable, that is, they cannot be expressed in a per-example manner, which hinders a straightforward application of M-estimation widely used in supervised learning. In this paper, we consider linear-fractional metrics, which are a family of classification performance metrics that encompasses many standard ones such as the F${}_β$-measure and Jaccard index, an…
RTL Nederland B.V.（RTL荷兰公司）；University of Amsterdam（阿姆斯特丹大学）
Multiclass multilabel classification is the task of attributing multiple labels to examples via predictions. Current models formulate a reduction of the multilabel setting into either multiple binary classifications or multiclass classification, allowing for the use of existing loss functions (sigmoid, cross-entropy, logistic, etc.). Multilabel classification reductions do not accommodate for the prediction of varying numbers of labels per example and the underlying losses are distant estimates of the performance metrics. We propose a loss function, sigmoidF1, which is an approximation of the …
Swanson School of Engineering, University of Pittsburgh（匹兹堡大学斯旺森工程学院）；Heinz College, Carnegie Mellon University（卡内基梅隆大学海因茨学院）；Meinig School of Biomedical Engineering, Cornell University（康奈尔大学迈尼格生物医学工程学院）
Objective: This study illustrates the ambiguity of ROC in evaluating two classifiers of 90-day LVAD mortality. This paper also introduces the precision recall curve (PRC) as a supplemental metric that is more representative of LVAD classifiers performance in predicting the minority class. Background: In the LVAD domain, the receiver operating characteristic (ROC) is a commonly applied metric of performance of classifiers. However, ROC can provide a distorted view of classifiers ability to predict short-term mortality due to the overwhelmingly greater proportion of patients who survive, i.e. im…
Kadir Has University（卡迪尔哈斯大学）；Huawei R&D Center（华为研发中心）
Classification of malware families is crucial for a comprehensive understanding of how they can infect devices, computers, or systems. Thus, malware identification enables security researchers and incident responders to take precautions against malware and accelerate mitigation. API call sequences made by malware are widely utilized features by machine and deep learning models for malware classification as these sequences represent the behavior of malware. However, traditional machine and deep learning models remain incapable of capturing sequence relationships between API calls. On the other …
Imbalanced class distribution is a common problem in a number of fields including medical diagnostics, fraud detection, and others. It causes bias in classification algorithms leading to poor performance on the minority class data. In this paper, we propose a novel method for balancing the class distribution in data through intelligent resampling of the minority class instances. The proposed method is based on generating new minority instances in the neighborhood of the existing minority points via a gamma distribution. Our method offers a natural and coherent approach to balancing the data. W…
Time Series Classification (TSC) has drawn a lot of attention in literature because of its broad range of applications for different domains, such as medical data mining, weather forecasting. Although TSC algorithms are designed for balanced datasets, most real-life time series datasets are imbalanced. The Skewed distribution is a problem for time series classification both in distance-based and feature-based algorithms under the condition of poor class separability. To address the imbalance problem, both sampling-based and algorithmic approaches are used in this paper. Different methods signi…
Imbalanced response variable distribution is a common occurrence in data science. In fields such as fraud detection, medical diagnostics, system intrusion detection and many others where abnormal behavior is rarely observed the data under study often features disproportionate target class distribution. One common way to combat class imbalance is through resampling the minority class to achieve a more balanced distribution. In this paper, we investigate the performance of the sampling method based on kernel density estimation (KDE). We believe that KDE offers a more natural way of generating ne…
Machine learning has automated much of financial fraud detection, notifying firms of, or even blocking, questionable transactions instantly. However, data imbalance starves traditionally trained models of the content necessary to detect fraud. This study examines three separate factors of credit card fraud detection via machine learning. First, it assesses the potential for different sampling methods, undersampling and Synthetic Minority Oversampling Technique (SMOTE), to improve algorithm performance in data-starved environments. Additionally, five industry-practical machine learning algorith…
China-EU Institute for Clean and Renewable Energy, Huazhong University of Science and Technology（华中科技大学中欧清洁与可再生能源学院）；School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（华中科技大学人工智能与自动化学院）；State Key Lab of Digital Manufacturing Equipment and Technology（数字制造装备与技术国家重点实验室）
This paper proposes a novel fault diagnosis approach based on generative adversarial networks (GAN) for imbalanced industrial time series where normal samples are much larger than failure cases. We combine a well-designed feature extractor with GAN to help train the whole network. Aimed at obtaining data distribution and hidden pattern in both original distinguishing features and latent space, the encoder-decoder-encoder three-sub-network is employed in GAN, based on Deep Convolution Generative Adversarial Networks (DCGAN) but without Tanh activation layer and only trained on normal samples. I…
School of Management Engineering, Ulsan National Institute of Science and Technology (UNIST)（蔚山科学技术院管理工程学院）
In clinical research, the lack of events of interest often necessitates imbalanced learning. One approach to resolve this obstacle is data integration or sharing, but due to privacy concerns neither is practical. Therefore, there is an increasing demand for a platform on which an analysis can be performed in a federated environment while maintaining privacy. However, it is quite challenging to develop a federated learning algorithm that can address both privacy-preserving and class imbalanced issues. In this study, we introduce a federated generative model learning platform for generating samp…
University of Michigan（密歇根大学）；Princeton University（普林斯顿大学）
We introduce UniLoss, a unified framework to generate surrogate losses for training deep networks with gradient descent, reducing the amount of manual design of task-specific surrogate losses. Our key observation is that in many cases, evaluating a model with a performance metric on a batch of examples can be refactored into four steps: from input to real-valued scores, from scores to comparisons of pairs of scores, from comparisons to binary variables, and from binary variables to the final performance metric. Using this refactoring we generate differentiable approximations for each non-diffe…
The present study introduces a method for improving the classification performance of imbalanced multiclass data streams from wireless body worn sensors. Data imbalance is an inherent problem in activity recognition caused by the irregular time distribution of activities, which are sequential and dependent on previous movements. We use conditional random fields (CRF), a graphical model for structured classification, to take advantage of dependencies between activities in a sequence. However, CRFs do not consider the negative effects of class imbalance during training. We propose a class-wise d…
University of Louisiana at Lafayette（路易斯安那大学拉斐特分校）
This study presents machine learning models that forecast and categorize lost circulation severity preemptively using a large class imbalanced drilling dataset. We demonstrate reproducible core techniques involved in tackling a large drilling engineering challenge utilizing easily interpretable machine learning approaches. We utilized a 65,000+ records data with class imbalance problem from Azadegan oilfield formations in Iran. Eleven of the dataset's seventeen parameters are chosen to be used in the classification of five lost circulation events. To generate classification models, we used six…
Luxembourg Institute of Science and Technology（卢森堡科学技术研究所）；University of Geneva（日内瓦大学）
We present novel active learning strategies dedicated to providing a solution to the cold start stage, i.e. initializing the classification of a large set of data with no attached labels. Moreover, proposed strategies are designed to handle an imbalanced context in which random selection is highly inefficient. Specifically, our active learning iterations address label scarcity and imbalance using element scores, combining information extracted from a clustering structure to a label propagation model. The strategy is illustrated by a case study on annotating Twitter content w.r.t. testimonies o…
Robust performance metrics for imbalanced classification problems
Philipps-Universität Marburg（马尔堡菲利普斯大学）；Karlsruher Institut für Technologie (KIT)（卡尔斯鲁厄理工学院）；Fachbereich Mathematik und Informatik（数学与信息学系）；Institut für Stochastik（统计研究所）
Convolutional neural networks (CNNs) have achieved great success in skin lesion classification. A balanced dataset is required to train a good model. However, due to the appearance of different skin lesions in practice, severe or even deadliest skin lesion types (e.g., melanoma) naturally have quite small amount represented in a dataset. In that, classification performance degradation occurs widely, it is significantly important to have CNNs that work well on class imbalanced skin lesion image dataset. In this paper, we propose SuperCon, a two-stage training strategy to overcome the class imba…
Université de Montréal（蒙特利尔大学）；University of Wrocław（弗罗茨瓦夫大学）
Often, the performance on a supervised machine learning task is evaluated with a emph{task loss} function that cannot be optimized directly. Examples of such loss functions include the classification error, the edit distance and the BLEU score. A common workaround for this problem is to instead optimize a emph{surrogate loss} function, such as for instance cross-entropy or hinge loss. In order for this remedy to be effective, it is important to ensure that minimization of the surrogate loss results in minimization of the task loss, a condition that we call emph{consistency with the task loss}.…
Exact Reformulation and Optimization for Direct Metric Optimization in Binary Imbalanced Classification
Department of Computer Science and Engineering, University of Minnesota, USA（明尼苏达大学计算机科学与工程系）；School of Statistics, University of Minnesota, USA（明尼苏达大学统计学系）；Department of Industrial Engineering and Operations Research, University of California, Berkeley, USA（加州大学伯克利分校工业工程与运筹学系）
The Dice score and Jaccard index are commonly used metrics for the evaluation of segmentation tasks in medical imaging. Convolutional neural networks trained for image segmentation tasks are usually optimized for (weighted) cross-entropy. This introduces an adverse discrepancy between the learning optimization objective (the loss) and the end target metric. Recent works in computer vision have proposed soft surrogates to alleviate this discrepancy and directly optimize the desired metric, either through relaxations (soft-Dice, soft-Jaccard) or submodular optimization (Lovász-softmax). The aim …
SCORE: A 1D Reparameterization Technique to Break Bayesian Optimization's Curse of Dimensionality
Laboratoire de Physique des Interfaces et des Couches Minces (LPICM)（界面与薄膜物理实验室（LPICM））；Ecole Polytechnique, Institut Polytechnique de Paris (IP Paris)（巴黎综合理工学院，巴黎理工学院（IP Paris））；Institut Photovoltaïque d’Île-de-France (IPVF)（法兰西岛光伏研究所（IPVF））
Bias-Corrected Data Synthesis for Imbalanced Learning
Department of Biostatistics & Bioinformatics, Duke University（生物统计学与生物信息学系，杜克大学）；Department of Electrical & Computer Engineering, Duke University（电气与计算机工程系，杜克大学）；Department of Statistics, Rutgers University（统计学系，罗格斯大学）；Department of Computer Science, Duke University（计算机科学系，杜克大学）
In this work, we investigated the application of score-based gradient learning in discriminative and generative classification settings. Score function can be used to characterize data distribution as an alternative to density. It can be efficiently learned via score matching, and used to flexibly generate credible samples to enhance discriminative classification quality, to recover density and to build generative classifiers. We analysed the decision theories involving score-based representations, and performed experiments on simulated and real-world datasets, demonstrating its effectiveness …
Texas A&M University（德州农工大学）；University of Washington（华盛顿大学）
Medical events of interest, such as mortality, often happen at a low rate in electronic medical records, as most admitted patients survive. Training models with this imbalance rate (class density discrepancy) may lead to suboptimal prediction. Traditionally this problem is addressed through ad-hoc methods such as resampling or reweighting but performance in many cases is still limited. We propose a framework for training models for this imbalance issue: 1) we first decouple the feature extraction and classification process, adjusting training batches separately for each component to mitigate b…
Information Technologies Institute（信息技术研究所）；Centre for Research and Technology—Hellas（希腊研究与技术中心）
Graph filters that transform prior node values to posterior scores via edge propagation often support graph mining tasks affecting humans, such as recommendation and ranking. Thus, it is important to make them fair in terms of satisfying statistical parity constraints between groups of nodes (e.g., distribute score mass between genders proportionally to their representation). To achieve this while minimally perturbing the original posteriors, we introduce a filter-aware universal approximation framework for posterior objectives. This defines appropriate graph neural networks trained at runtime…
