We show when maximizing a properly defined $f$-divergence measure with respect to a classifier's predictions and the supervised labels is robust with label noise. Leveraging its variational form, we derive a nice decoupling property for a family of $f$-divergence measures when label noise presents, where the divergence is shown to be a linear combination of the variational difference defined on the clean distribution and a bias term introduced due to the noise. The above derivation helps us analyze the robustness of different $f$-divergence functions. With established robustness, this family of $f$-divergence functions arises as useful metrics for the problem of learning with noisy labels, which do not require the specification of the labels' noise rate. When they are possibly not robust, we propose fixes to make them so. In addition to the analytical results, we present thorough experimental evidence. Our code is available at https://github.com/UCSC-REAL/Robust-f-divergence-measures.
Learning to Forget with Information Divergence Reweighted Objectives for Noisy Labels
Department of Mathematics（数学系）；Texas State University（德克萨斯州立大学）；School of Information Systems and Management（信息系统与管理学院）；University of South Florida（佛罗里达州立大学）
University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）；Foundation for Research and Technology, Hellas（希腊研究与技术基金会）
Variational representations of divergences and distances between high-dimensional probability distributions offer significant theoretical insights and practical advantages in numerous research areas. Recently, they have gained popularity in machine learning as a tractable and scalable approach for training probabilistic models and for statistically differentiating between data distributions. Their advantages include: 1) They can be estimated from data as statistical averages. 2) Such representations can leverage the ability of neural networks to efficiently approximate optimal solutions in fun…
Prior works have found it beneficial to combine provably noise-robust loss functions e.g., mean absolute error (MAE) with standard categorical loss function e.g. cross entropy (CE) to improve their learnability. Here, we propose to use Jensen-Shannon divergence as a noise-robust loss function and show that it interestingly interpolate between CE and MAE with a controllable mixing parameter. Furthermore, we make a crucial observation that CE exhibit lower consistency around noisy data points. Based on this observation, we adopt a generalized version of the Jensen-Shannon divergence for multiple…
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Transfer learning has been successfully applied across many high-impact applications. However, most existing work focuses on the static transfer learning setting, and very little is devoted to modeling the time evolving target domain, such as the online reviews for movies. To bridge this gap, in this paper, we study a novel continuous transfer learning setting with a time evolving target domain. One major challenge associated with continuous transfer learning is the potential occurrence of negative transfer as the target domain evolves over time. To address this challenge, we propose a novel l…
Fraunhofer-Institute for Intelligent Analysis and Information Systems (IAIS)（弗劳恩霍夫智能分析与信息系统研究所（IAIS））；University of Bonn（波恩大学）
The concept of Label Distribution Learning (LDL) is a technique to stabilize classification and regression problems with ambiguous and/or imbalanced labels. A prototypical use-case of LDL is human age estimation based on profile images. Regarding this regression problem, a so called Deep Label Distribution Learning (DLDL) method has been developed. The main idea is the joint regression of the label distribution and its expectation value. However, the original DLDL method uses loss components with different mathematical motivation and, thus, different scales, which is why the use of a hyperpara…
This paper builds upon the work of Pfau (2013), which generalized the bias variance tradeoff to any Bregman divergence loss function. Pfau (2013) showed that for Bregman divergences, the bias and variances are defined with respect to a central label, defined as the mean of the label variable, and a central prediction, of a more complex form. We show that, similarly to the label, the central prediction can be interpreted as the mean of a random variable, where the mean operates in a dual space defined by the loss function itself. Viewing the bias-variance tradeoff through operations taken in du…
Tsinghua University（清华大学）；Jiangsu Normal University（江苏师范大学）
Various evaluation measures have been developed for multi-label classification, including Hamming Loss (HL), Subset Accuracy (SA) and Ranking Loss (RL). However, there is a gap between empirical results and the existing theories: 1) an algorithm often empirically performs well on some measure(s) while poorly on others, while a formal theoretical analysis is lacking; and 2) in small label space cases, the algorithms optimizing HL often have comparable or even better performance on the SA measure than those optimizing SA directly, while existing theoretical results show that SA and HL are confli…
Semi-supervised learning has received attention from researchers, as it allows one to exploit the structure of unlabeled data to achieve competitive classification results with much fewer labels than supervised approaches. The Local and Global Consistency (LGC) algorithm is one of the most well-known graph-based semi-supervised (GSSL) classifiers. Notably, its solution can be written as a linear combination of the known labels. The coefficients of this linear combination depend on a parameter $α$, determining the decay of the reward over time when reaching labeled vertices in a random walk. In…
Online continual learning, especially when task identities and task boundaries are unavailable, is a challenging continual learning setting. One representative kind of methods for online continual learning is replay-based methods, in which a replay buffer called memory is maintained to keep a small part of past samples for overcoming catastrophic forgetting. When tackling with online continual learning, most existing replay-based methods focus on single-label problems in which each sample in the data stream has only one label. But multi-label problems may also happen in the online continual le…
Enhanced ECG Arrhythmia Detection Accuracy by Optimizing Divergence-Based Data Fusion
Aerie Intelligent Technology Corp（艾瑞智能科技公司）；People’s Hospital of Shenzhen Baoan District Shenzhen（深圳市宝安区人民医院）；The Second Affiliated Hospital of Shenzhen University（深圳大学第二附属医院）
University of Maryland（马里兰大学）；Johns Hopkins University（约翰斯·霍普金斯大学）
Numerous large-scale chest x-ray datasets have spearheaded expert-level detection of abnormalities using deep learning. However, these datasets focus on detecting a subset of disease labels that could be present, thus making them distributed and non-iid with partial labels. Recent literature has indicated the impact of batch normalization layers on the convergence of federated learning due to domain shift associated with non-iid data with partial labels. To that end, we propose FedFBN, a federated learning framework that draws inspiration from transfer learning by using pretrained networks as …
Optimizing Feature Selection for Binary Classification with Noisy Labels: A Genetic Algorithm Approach
A. I. Virtanen Institute for Molecular Sciences, University of Eastern Finland（A.I. Virtanen分子科学研究所，东芬兰大学）；Department of Computer Science, Aalto University（计算机科学系，艾尔托大学）；Institute of Biomedicine, University of Eastern Finland（生物医学研究所，东芬兰大学）
UIUC（伊利诺伊大学厄巴纳-香槟分校）；Google Research USA（谷歌美国研究院）；Google Research Accra（谷歌阿克拉研究院）
We consider learning to optimize a classification metric defined by a black-box function of the confusion matrix. Such black-box learning settings are ubiquitous, for example, when the learner only has query access to the metric of interest, or in noisy-label and domain adaptation applications where the learner must evaluate the metric via performance evaluation using a small validation sample. Our approach is to adaptively learn example weights on the training dataset such that the resulting weighted objective best approximates the metric on the validation sample. We show how to model and est…
MatchNAS: Optimizing Edge AI in Sparse-Label Data Contexts via Automating Deep Neural Network Porting for Mobile Deployment
School of Computer Science and Engineering, University of New South Wales（新南威尔士大学计算机科学与工程学院）；Faculty of Engineering & Information Technology, University of Technology Sydney（悉尼科技大学工程与信息技术学院）；CSIRO’s Data61（澳大利亚联邦科学与工业研究组织Data61）；University of New South Wales（新南威尔士大学）
We study the problem of agnostically learning homogeneous halfspaces in the distribution-specific PAC model. For a broad family of structured distributions, including log-concave distributions, we show that non-convex SGD efficiently converges to a solution with misclassification error $O(\opt)+\eps$, where $\opt$ is the misclassification error of the best-fitting halfspace. In sharp contrast, we show that optimizing any convex surrogate inherently leads to misclassification error of $ω(\opt)$, even under Gaussian marginals.
Optimizing Non-decomposable Measures with Deep Networks
University of Oxford（牛津大学）；IIT Kanpur（印度理工学院坎普尔分校）；Qatar Computing Research Institute（卡塔尔计算研究所）；Istituto di Scienza e Tecnologia dell’Informazione（意大利信息科学与技术研究所）
Many problems in machine learning can be formulated as optimizing a convex functional over a vector space of measures. This paper studies the convergence of the mirror descent algorithm in this infinite-dimensional setting. Defining Bregman divergences through directional derivatives, we derive the convergence of the scheme for relatively smooth and convex pairs of functionals. Such assumptions allow to handle non-smooth functionals such as the Kullback--Leibler (KL) divergence. Applying our result to joint distributions and KL, we show that Sinkhorn's primal iterations for entropic optimal tr…
The Multiplex Classification Framework: optimizing multi-label classifiers through problem transformation, ontology engineering, and model ensembling
Universidad Nacional del Sur（国立南方大学）；National Scientific and Technological Research Council of Argentina (CONICET)（阿根廷国家科学技术研究委员会（CONICET））；Centro Nacional Patagónico (CENPAT-CONICET)（巴塔哥尼亚国家中心（CENPAT-CONICET））
Mitsubishi Electric Research Labs (MERL)（三菱电机研究实验室）；Australian Centre for Robotic Vision, Australian National University（澳大利亚国立大学澳大利亚机器人视觉中心）；University of Minnesota（明尼苏达大学）；Research School of Engineering, The Australian National University（澳大利亚国立大学工程研究学院）；Monash University（莫纳什大学）；Data61-CSIRO（澳大利亚联邦科学与工业研究组织数据61）
Representations in the form of Symmetric Positive Definite (SPD) matrices have been popularized in a variety of visual learning applications due to their demonstrated ability to capture rich second-order statistics of visual data. There exist several similarity measures for comparing SPD matrices with documented benefits. However, selecting an appropriate measure for a given problem remains a challenge and in most cases, is the result of a trial-and-error process. In this paper, we propose to learn similarity measures in a data-driven manner. To this end, we capitalize on the αβ-log-det diverg…
Optimizing Federated Learning by Entropy-Based Client Selection
Department of Artificial Intelligence, Fraunhofer Heinrich-Hertz-Institute（人工智能系，弗劳恩霍夫海因里希·赫兹研究所）；Department of Mathematics, Technische Universität Berlin（数学系，柏林技术大学）；Department of Electrical Engineering and Computer Science, Technische Universität Berlin（电气工程与计算机科学系，柏林技术大学）；BIFOLD – Berlin Institute for the Foundations of Learning（BIFOLD——柏林学习与数据基础研究所）
