The Wasserstein distance provides a notion of dissimilarities between probability measures, which has recent applications in learning of structured data with varying size such as images and text documents. In this work, we study the $k$-nearest neighbor classifier ($k$-NN) of probability measures under the Wasserstein distance. We show that the $k$-NN classifier is not universally consistent on the space of measures supported in $(0,1)$. As any Euclidean ball contains a copy of $(0,1)$, one should not expect to obtain universal consistency without some restriction on the base metric space, or the Wasserstein space itself. To this end, via the notion of $σ$-finite metric dimension, we show that the $k$-NN classifier is universally consistent on spaces of measures supported in a $σ$-uniformly discrete set. In addition, by studying the geodesic structures of the Wasserstein spaces for $p=1$ and $p=2$, we show that the $k$-NN classifier is universally consistent on the space of measures supported on a finite set, the space of Gaussian measures, and the space of measures with densities expressed as finite wavelet series.
Artificial and Natural Intelligence Toulouse Institute (ANITI)（图卢兹人工智能与自然智能研究所）；Institut de Recherche Technologique (IRT) Saint Exupéry（圣埃克苏佩里技术研究所）
The increasingly common use of neural network classifiers in industrial and social applications of image analysis has allowed impressive progress these last years. Such methods are however sensitive to algorithmic bias, i.e. to an under- or an over-representation of positive predictions or to higher prediction errors in specific subgroups of images. We then introduce in this paper a new method to temper the algorithmic bias in Neural-Network based classifiers. Our method is Neural-Network architecture agnostic and scales well to massive training sets of images. It indeed only overloads the los…
University of South Brittany（南布列塔尼大学）；CNRS（法国国家科学研究中心）；IRISA（雷恩第一大学计算机科学与应用数学研究所）
Adversarial examples are a hot topic due to their abilities to fool a classifier's prediction. There are two strategies to create such examples, one uses the attacked classifier's gradients, while the other only requires access to the clas-sifier's prediction. This is particularly appealing when the classifier is not full known (black box model). In this paper, we present a new method which is able to generate natural adversarial examples from the true data following the second paradigm. Based on Generative Adversarial Networks (GANs) [5], it reweights the true data empirical distribution to e…
$k$-Nearest Neighbors in Gromov--Wasserstein Space
University of North Carolina at Chapel Hill, Department of Mathematics（北卡罗来纳大学教堂山分校数学系）；University of North Carolina at Chapel Hill, Department of Statistics and Operations Research（北卡罗来纳大学教堂山分校统计与运筹学系）
Indian Statistical Institute, Kolkata（印度统计研究所（加尔各答））
The introduction of Variational Autoencoders (VAE) has been marked as a breakthrough in the history of representation learning models. Besides having several accolades of its own, VAE has successfully flagged off a series of inventions in the form of its immediate successors. Wasserstein Autoencoder (WAE), being an heir to that realm carries with it all of the goodness and heightened generative promises, matching even the generative adversarial networks (GANs). Needless to say, recent years have witnessed a remarkable resurgence in statistical analyses of the GANs. Similar examinations for Aut…
Stanford University（斯坦福大学）；Seoul National University（首尔国立大学）
Wasserstein distributionally robust optimization (WDRO) attempts to learn a model that minimizes the local worst-case risk in the vicinity of the empirical data distribution defined by Wasserstein ball. While WDRO has received attention as a promising tool for inference since its introduction, its theoretical understanding has not been fully matured. Gao et al. (2017) proposed a minimizer based on a tractable approximation of the local worst-case risk, but without showing risk consistency. In this paper, we propose a minimizer based on a novel approximation theorem and provide the correspondin…
Carnegie Mellon University（卡内基梅隆大学）；Bosch Center for Artificial Intelligence（博世人工智能中心）
A rapidly growing area of work has studied the existence of adversarial examples, datapoints which have been perturbed to fool a classifier, but the vast majority of these works have focused primarily on threat models defined by $\ell_p$ norm-bounded perturbations. In this paper, we propose a new threat model for adversarial attacks based on the Wasserstein distance. In the image classification setting, such distances measure the cost of moving pixel mass, which naturally cover "standard" image manipulations such as scaling, rotation, translation, and distortion (and can potentially be applied…
University Bretagne Sud（布列塔尼南部大学）；CNRS（法国国家科学研究中心）；INRIA（法国国家信息与自动化研究所）；IRISA（法国信息与自动化系统研究所）；InterDigital R&D（InterDigital研发中心）；Université de Paris（巴黎大学）；Wageningen University（瓦赫宁根大学）；École Polytechnique（巴黎综合理工学院）；Ecole Polytechnique Fédérale de Lausanne（洛桑联邦理工学院）
Noisy labels often occur in vision datasets, especially when they are obtained from crowdsourcing or Web scraping. We propose a new regularization method, which enables learning robust classifiers in presence of noisy data. To achieve this goal, we propose a new adversarial regularization scheme based on the Wasserstein distance. Using this distance allows taking into account specific relations between classes by leveraging the geometric properties of the labels space. Our Wasserstein Adversarial Regularization (WAR) encodes a selective regularization, which promotes smoothness of the classifi…
University of Central Florida（中佛罗里达大学）；Beijing Jiaotong University（北京交通大学）；Tencent AI Lab（腾讯人工智能实验室）
Despite being impactful on a variety of problems and applications, the generative adversarial nets (GANs) are remarkably difficult to train. This issue is formally analyzed by \cite{arjovsky2017towards}, who also propose an alternative direction to avoid the caveats in the minmax two-player training of GANs. The corresponding algorithm, called Wasserstein GAN (WGAN), hinges on the 1-Lipschitz continuity of the discriminator. In this paper, we propose a novel approach to enforcing the Lipschitz continuity in the training procedure of WGANs. Our approach seamlessly connects WGAN with one of the …
Multithreshold Entropy Linear Classifier (MELC) is a recent classifier idea which employs information theoretic concept in order to create a multithreshold maximum margin model. In this paper we analyze its consistency over multithreshold linear models and show that its objective function upper bounds the amount of misclassified points in a similar manner like hinge loss does in support vector machines. For further confirmation we also conduct some numerical experiments on five datasets.
The Nearest subspace classifier (NSS) finds an estimation of the underlying subspace within each class and assigns data points to the class that corresponds to its nearest subspace. This paper mainly studies how well NSS can be generalized to new samples. It is proved that NSS is strongly consistent under certain assumptions. For completeness, NSS is evaluated through experiments on various simulated and real data sets, in comparison with some other linear model based classifiers. It is also shown that NSS can obtain effective classification results and is very efficient, especially for large …
Skolkovo Institute of Science and Technology（斯科尔科沃科学技术研究院）；Massachusetts Institute of Technology（麻省理工学院）
Wasserstein barycenters provide a geometric notion of the weighted average of probability measures based on optimal transport. In this paper, we present a scalable algorithm to compute Wasserstein-2 barycenters given sample access to the input measures, which are not restricted to being discrete. While past approaches rely on entropic or quadratic regularization, we employ input convex neural networks and cycle-consistency regularization to avoid introducing bias. As a result, our approach does not resort to minimax optimization. We provide theoretical analysis on error bounds as well as empir…
Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院）
A recent technique of randomized smoothing has shown that the worst-case (adversarial) $\ell_2$-robustness can be transformed into the average-case Gaussian-robustness by "smoothing" a classifier, i.e., by considering the averaged prediction over Gaussian noise. In this paradigm, one should rethink the notion of adversarial robustness in terms of generalization ability of a classifier under noisy observations. We found that the trade-off between accuracy and certified robustness of smoothed classifiers can be greatly controlled by simply regularizing the prediction consistency over noise. This…
We propose a distributionally robust logistic regression model with an unfairness penalty that prevents discrimination with respect to sensitive attributes such as gender or ethnicity. This model is equivalent to a tractable convex optimization problem if a Wasserstein ball centered at the empirical distribution on the training data is used to model distributional uncertainty and if a new convex unfairness measure is used to incentivize equalized opportunities. We demonstrate that the resulting classifier improves fairness at a marginal loss of predictive accuracy on both synthetic and real da…
The cellular tree classifier model addresses a fundamental problem in the design of classifiers for a parallel or distributed computing world: Given a data set, is it sufficient to apply a majority rule for classification, or shall one split the data into two or more parts and send each part to a potentially different computer (or cell) for further processing? At first sight, it seems impossible to define with this paradigm a consistent classifier as no cell knows the "original data size", $n$. However, we show that this is not so by exhibiting two different consistent classifiers. The consist…
Department of Applied Mathematics and Theoretical Physics, The University of Cambridge（应用数学与理论物理系，剑桥大学）；Department of Medicine, The University of Cambridge（医学系，剑桥大学）；Department of Statistics, The University of Warwick（统计系，沃里克大学）
University of Hull（赫尔大学）；Fraunhofer Institute for Experimental Software Engineering, IESE（弗劳恩霍夫实验软件工程研究所（IESE））；Fraunhofer-Gesellschaft（弗劳恩霍夫协会）；APD Communications（APD通信公司）；Nuremberg Tech（纽伦堡应用技术大学）
Ensuring safety and explainability of machine learning (ML) is a topic of increasing relevance as data-driven applications venture into safety-critical application domains, traditionally committed to high safety standards that are not satisfied with an exclusive testing approach of otherwise inaccessible black-box systems. Especially the interaction between safety and security is a central challenge, as security violations can lead to compromised safety. The contribution of this paper to addressing both safety and security within a single concept of protection applicable during the operation o…
University of Toronto（多伦多大学）；Lawrence Berkeley National Laboratory（劳伦斯伯克利国家实验室）；California Institute of Technology（加州理工学院）；Inria Paris, École des Ponts（法国国家信息与自动化研究所巴黎分所、巴黎桥路学院）
Projected Wasserstein-Fisher-Rao Flows for Hawkes Background-Intensity Estimation
Facultad de Ciencias, Universidad Autónoma de México（墨西哥国立自治大学理学院）；Department of Probability and Statistics, Centro de Investigación en Matemáticas A.C.（数学研究中心概率与统计系）
Institute of Computer Science, Polish Academy of Sciences（波兰科学院计算机科学研究所）
Classifier chains are popular and effective method to tackle a multi-label classification problem. The aim of this paper is to study the asymptotic properties of the chain model in which the conditional probabilities are of the logistic form. In particular we find conditions on the number of labels and the distribution of feature vector under which the estimated mode of the joint distribution of labels converges to the true mode. Best of our knowledge, this important issue has not yet been studied in the context of multi-label learning. We also investigate how the order of model building in a …
We present a novel class of projected methods, to perform statistical analysis on a data set of probability distributions on the real line, with the 2-Wasserstein metric. We focus in particular on Principal Component Analysis (PCA) and regression. To define these models, we exploit a representation of the Wasserstein space closely related to its weak Riemannian structure, by mapping the data to a suitable linear space and using a metric projection operator to constrain the results in the Wasserstein space. By carefully choosing the tangent point, we are able to derive fast empirical methods, e…
Deep Copula Classifier: Theory, Consistency, and Empirical Evaluation
Department of Mathematics, University of Louisiana at Lafayette（路易斯安那大学拉福莱分校数学系）；Department of Statistics, Formerly of Presidency College, Kolkata, India（印度加尔各答 Presidency College 统计学系）
This manuscript studies statistical properties of linear classifiers obtained through minimization of an unregularized convex risk over a finite sample. Although the results are explicitly finite-dimensional, inputs may be passed through feature maps; in this way, in addition to treating the consistency of logistic regression, this analysis also handles boosting over a finite weak learning class with, for instance, the exponential, logistic, and hinge losses. In this finite-dimensional setting, it is still possible to fit arbitrary decision boundaries: scaling the complexity of the weak learni…
