This paper presents a proposal of a faster Wasserstein $k$-means algorithm for histogram data by reducing Wasserstein distance computations and exploiting sparse simplex projection. We shrink data samples, centroids, and the ground cost matrix, which leads to considerable reduction of the computations used to solve optimal transport problems without loss of clustering quality. Furthermore, we dynamically reduced the computational complexity by removing lower-valued data samples and harnessing sparse simplex projection while keeping the degradation of clustering quality lower. We designate this proposed algorithm as sparse simplex projection based Wasserstein $k$-means, or SSPW $k$-means. Numerical evaluations conducted with comparison to results obtained using Wasserstein $k$-means algorithm demonstrate the effectiveness of the proposed SSPW $k$-means for real-world datasets
Seeking informative projecting directions has been an important task in utilizing sliced Wasserstein distance in applications. However, finding these directions usually requires an iterative optimization procedure over the space of projecting directions, which is computationally expensive. Moreover, the computational issue is even more severe in deep learning applications, where computing the distance between two mini-batch probability measures is repeated several times. This nested loop has been one of the main challenges that prevent the usage of sliced Wasserstein distances based on good pr…
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）
Sliced Wasserstein (SW) distance has been widely used in different application scenarios since it can be scaled to a large number of supports without suffering from the curse of dimensionality. The value of sliced Wasserstein distance is the average of transportation cost between one-dimensional representations (projections) of original measures that are obtained by Radon Transform (RT). Despite its efficiency in the number of supports, estimating the sliced Wasserstein requires a relatively large number of projections in high-dimensional settings. Therefore, for applications where the number …
An independence model for discrete random variables is a Segre-Veronese variety in a probability simplex. Any metric on the set of joint states of the random variables induces a Wasserstein metric on the probability simplex. The unit ball of this polyhedral norm is dual to the Lipschitz polytope. Given any data distribution, we seek to minimize its Wasserstein distance to a fixed independence model. The solution to this optimization problem is a piecewise algebraic function of the data. We compute this function explicitly in small instances, we examine its combinatorial structure and algebraic…
Georgia Institute of Technology（佐治亚理工学院）；East China Normal University（华东师范大学）
Wasserstein distance plays increasingly important roles in machine learning, stochastic programming and image processing. Major efforts have been under way to address its high computational complexity, some leading to approximate or regularized variations such as Sinkhorn distance. However, as we will demonstrate, regularized variations with large regularization parameter will degradate the performance in several important machine learning applications, and small regularization parameter will fail due to numerical stability issues with existing algorithms. We address this challenge by developi…
University of California, Berkeley（加州大学伯克利分校）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；University of Texas, Austin（德克萨斯大学奥斯汀分校）；CREST - ENSAE（法国国家统计与经济管理学院经济与统计研究中心）；Google Brain（谷歌大脑）
Projection robust Wasserstein (PRW) distance, or Wasserstein projection pursuit (WPP), is a robust variant of the Wasserstein distance. Recent work suggests that this quantity is more robust than the standard Wasserstein distance, in particular when comparing probability measures in high-dimensions. However, it is ruled out for practical application because the optimization model is essentially non-convex and non-smooth which makes the computation intractable. Our contribution in this paper is to revisit the original motivation behind WPP/PRW, but take the hard route of showing that, despite i…
The Chinese University of Hong Kong（香港中文大学）；Nanjing University（南京大学）
Wasserstein \textbf{D}istributionally \textbf{R}obust \textbf{O}ptimization (DRO) is concerned with finding decisions that perform well on data that are drawn from the worst-case probability distribution within a Wasserstein ball centered at a certain nominal distribution. In recent years, it has been shown that various DRO formulations of learning models admit tractable convex reformulations. However, most existing works propose to solve these convex reformulations by general-purpose solvers, which are not well-suited for tackling large-scale problems. In this paper, we focus on a family of W…
VinAI Research（VinAI研究院）；University of Texas, Austin（得克萨斯大学奥斯汀分校）
Sliced-Wasserstein distance (SW) and its variant, Max Sliced-Wasserstein distance (Max-SW), have been used widely in the recent years due to their fast computation and scalability even when the probability measures lie in a very high dimensional space. However, SW requires many unnecessary projection samples to approximate its value while Max-SW only uses the most important projection, which ignores the information of other useful directions. In order to account for these weaknesses, we propose a novel distance, named Distributional Sliced-Wasserstein distance (DSW), that finds an optimal dist…
The Wasserstein distance has become increasingly important in machine learning and deep learning. Despite its popularity, the Wasserstein distance is hard to approximate because of the curse of dimensionality. A recently proposed approach to alleviate the curse of dimensionality is to project the sampled data from the high dimensional probability distribution onto a lower-dimensional subspace, and then compute the Wasserstein distance between the projected data. However, this approach requires to solve a max-min problem over the Stiefel manifold, which is very challenging in practice. The only…
Computing optimal transport (OT) between measures in high dimensions is doomed by the curse of dimensionality. A popular approach to avoid this curse is to project input measures on lower-dimensional subspaces (1D lines in the case of sliced Wasserstein distances), solve the OT problem between these reduced measures, and settle for the Wasserstein distance between these reductions, rather than that between the original measures. This approach is however difficult to extend to the case in which one wants to compute an OT map (a Monge map) between the original measures. Since computations are ca…
University of California, Berkeley（加州大学伯克利分校）；CREST - ENSAE（法国国家统计与经济管理学院 Crest 实验室）；Google Brain（谷歌大脑）
Optimal transport (OT) distances are increasingly used as loss functions for statistical inference, notably in the learning of generative models or supervised learning. Yet, the behavior of minimum Wasserstein estimators is poorly understood, notably in high-dimensional regimes or under model misspecification. In this work we adopt the viewpoint of projection robust (PR) OT, which seeks to maximize the OT cost between two measures by choosing a $k$-dimensional subspace onto which they can be projected. Our first contribution is to establish several fundamental statistical properties of PR Wass…
Wasserstein Discriminant Analysis (WDA) is a new supervised method that can improve classification of high-dimensional data by computing a suitable linear map onto a lower dimensional subspace. Following the blueprint of classical Linear Discriminant Analysis (LDA), WDA selects the projection matrix that maximizes the ratio of two quantities: the dispersion of projected points coming from different classes, divided by the dispersion of projected points coming from the same class. To quantify dispersion, WDA uses regularized Wasserstein distances, rather than cross-variance measures which have …
University College London（伦敦大学学院）；Facebook AI Research（脸书人工智能研究院）；University of Cambridge（剑桥大学）；University of California, Berkeley（加州大学伯克利分校）；Imperial College London（帝国理工学院）
Multi-marginal optimal transport enables one to compare multiple probability measures, which increasingly finds application in multi-task learning problems. One practical limitation of multi-marginal transport is computational scalability in the number of measures, samples and dimensionality. In this work, we propose a multi-marginal optimal transport paradigm based on random one-dimensional projections, whose (generalized) distance we term the sliced multi-marginal Wasserstein distance. To construct this distance, we introduce a characterization of the one-dimensional multi-marginal Kantorovi…
Adversarial examples are crafted by adding indistinguishable perturbations to normal examples in order to fool a well-trained deep learning model to misclassify. In the context of computer vision, this notion of indistinguishability is typically bounded by $L_{\infty}$ or other norms. However, these norms are not appropriate for measuring indistinguishiability for time series data. In this work, we propose adversarial examples in the Wasserstein space for time series data for the first time and utilize Wasserstein distance to bound the perturbation between normal examples and adversarial examp…
Fused Gromov-Wasserstein Distance with Feature Selection
School of Data Science and Society（数据科学与社会学院）；Division of Computational Social Science（计算社会科学系）；University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Courant Institute（Courant研究所）；Department of Mathematics（数学系）；New York University（纽约大学）；Duke University（杜克大学）
Leveraging Optimal Transport via Projections on Subspaces for Machine Learning Applications
Université de Bretagne Sud（南布列塔尼大学）；Laboratoire de Mathématiques de Bretagne Atlantique（大西洋布列塔尼数学实验室）；CNRS（法国国家科学研究中心）；École Normale Supérieure（巴黎高等师范学院）；Technische Universität Berlin（柏林工业大学）；Université Paris Cité（巴黎西岱大学）；Sony Computer Science Laboratories Inc（索尼计算机科学实验室公司）；Harvard SEAS（哈佛大学工程与应用科学学院）；IMT Atlantique（大西洋国立高等工业技术与矿业学校）
In generative modeling, the Wasserstein distance (WD) has emerged as a useful metric to measure the discrepancy between generated and real data distributions. Unfortunately, it is challenging to approximate the WD of high-dimensional distributions. In contrast, the sliced Wasserstein distance (SWD) factorizes high-dimensional distributions into their multiple one-dimensional marginal distributions and is thus easier to approximate. In this paper, we introduce novel approximations of the primal and dual SWD. Instead of using a large number of random projections, as it is done by conventional SW…
Most learning methods with rank or sparsity constraints use convex relaxations, which lead to optimization with the nuclear norm or the $\ell_1$-norm. However, several important learning applications cannot benefit from this approach as they feature these convex norms as constraints in addition to the non-convex rank and sparsity constraints. In this setting, we derive efficient sparse projections onto the simplex and its extension, and illustrate how to use them to solve high-dimensional learning problems in quantum tomography, sparse density estimation and portfolio selection with non-convex…
Stanford University（斯坦福大学）；Singapore University of Technology and Design（新加坡科技设计大学）；VinAI Research（VinAI研究院）
We consider statistical methods which invoke a min-max distributionally robust formulation to extract good out-of-sample performance in data-driven optimization and learning problems. Acknowledging the distributional uncertainty in learning from limited samples, the min-max formulations introduce an adversarial inner player to explore unseen covariate data. The resulting Distributionally Robust Optimization (DRO) formulations, which include Wasserstein DRO formulations (our main focus), are specified using optimal transportation phenomena. Upon describing how these infinite-dimensional min-max…
We provide an elementary proof of a simple, efficient algorithm for computing the Euclidean projection of a point onto the probability simplex. We also show an application in Laplacian K-modes clustering.
School of Mathematics and Maxwell Institute for Mathematical Sciences, The University of Edinburgh（爱丁堡大学数学学院和麦克斯韦数学科学研究所）；Department of Applied Mathematics and Theoretical Physics, University of Cambridge（剑桥大学应用数学与理论物理系）
Relative Wasserstein Angle and the Problem of the $W_2$-Nearest Gaussian Distribution
Department of Computer Science, George Washington University, Washington D.C., USA（计算机科学系，乔治·华盛顿大学，华盛顿特区，美国）；MAE, George Washington University, Washington D.C., USA（机械工程与应用科学系，乔治·华盛顿大学，华盛顿特区，美国）
Sliced Wasserstein Steering between Gaussian Measures
Department of Information Physics and Computing, The University of Tokyo（信息物理与计算系，东京大学）；Department of Decision and Control Systems and Department of Mathematics, KTH Royal Institute of Technology（决策与控制系统系和数学系，皇家理工学院）
Toyota Technological Institute at Chicago（芝加哥丰田理工学院）；National University of Singapore（新加坡国立大学）
We provide a simple and efficient algorithm for computing the Euclidean projection of a point onto the capped simplex---a simplex with an additional uniform bound on each coordinate---together with an elementary proof. Both the MATLAB and C++ implementations of the proposed algorithm can be downloaded at https://eng.ucmerced.edu/people/wwang5.
Collecting and aggregating information from several probability measures or histograms is a fundamental task in machine learning. One of the popular solution methods for this task is to compute the barycenter of the probability measures under the Wasserstein metric. However, approximating the Wasserstein barycenter is numerically challenging because of the curse of dimensionality. This paper proposes the projection robust Wasserstein barycenter (PRWB) that has the potential to mitigate the curse of dimensionality. Since PRWB is numerically very challenging to solve, we further propose a relaxe…
École Polytechnique（巴黎综合理工学院）；University of Texas, Austin（德克萨斯大学奥斯汀分校）；Hanoi University of Science and Technology（河内科技大学）
Generalized sliced Wasserstein distance is a variant of sliced Wasserstein distance that exploits the power of non-linear projection through a given defining function to better capture the complex structures of the probability distributions. Similar to sliced Wasserstein distance, generalized sliced Wasserstein is defined as an expectation over random projections which can be approximated by the Monte Carlo method. However, the complexity of that approximation can be expensive in high-dimensional settings. To that end, we propose to form deterministic and fast approximations of the generalized…
LTCI, Télécom Paris, Institut Polytechnique de Paris（巴黎电信学院，巴黎综合理工学院）；ESSEC Business School（ESSEC商学院）
The Sliced-Wasserstein distance (SW) is being increasingly used in machine learning applications as an alternative to the Wasserstein distance and offers significant computational and statistical benefits. Since it is defined as an expectation over random projections, SW is commonly approximated by Monte Carlo. We adopt a new perspective to approximate SW by making use of the concentration of measure phenomenon: under mild assumptions, one-dimensional projections of a high-dimensional random vector are approximately Gaussian. Based on this observation, we develop a simple deterministic approxi…
University of Surrey（萨里大学）；University of Edinburgh（爱丁堡大学）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）
While theoretically appealing, the application of the Wasserstein distance to large-scale machine learning problems has been hampered by its prohibitive computational cost. The sliced Wasserstein distance and its variants improve the computational efficiency through the random projection, yet they suffer from low accuracy if the number of projections is not sufficiently large, because the majority of projections result in trivially small values. In this work, we propose a new family of distance metrics, called augmented sliced Wasserstein distances (ASWDs), constructed by first mapping samples…
Inverse problems consist of recovering a signal from a collection of noisy measurements. These are typically cast as optimization problems, with classic approaches using a data fidelity term and an analytic regularizer that stabilizes recovery. Recent Plug-and-Play (PnP) works propose replacing the operator for analytic regularization in optimization methods by a data-driven denoiser. These schemes obtain state of the art results, but at the cost of limited theoretical guarantees. To bridge this gap, we present a new algorithm that takes samples from the manifold of true data as input and outp…
We consider the general dimensionality reduction problem of locating in a high-dimensional data cloud, a $k$-dimensional non-Gaussian subspace of interesting features. We use a projection pursuit approach -- we search for mutually orthogonal unit directions which maximise the 2-Wasserstein distance of the empirical distribution of data-projections along these directions from a standard Gaussian. Under a generative model, where there is a underlying (unknown) low-dimensional non-Gaussian subspace, we prove rigorous statistical guarantees on the accuracy of approximating this unknown subspace by…
RIKEN Interdisciplinary Theoretical and Mathematical Sciences Program（理化学研究所跨学科理论与数学科学项目）；RIKEN Center for Advanced Intelligence Project（理化学研究所先进智能项目中心）；Keio University（庆应义塾大学）；Fujitsu Limited（富士通株式会社）；Tokyo Institute of Technology（东京工业大学）；KLab Inc.（KLab株式会社）
Bézier simplex fitting algorithms have been recently proposed to approximate the Pareto set/front of multi-objective continuous optimization problems. These new methods have shown to be successful at approximating various shapes of Pareto sets/fronts when sample points exactly lie on the Pareto set/front. However, if the sample points scatter away from the Pareto set/front, those methods often likely suffer from over-fitting. To overcome this issue, in this paper, we extend the Bézier simplex model to a probabilistic one and propose a new learning algorithm of it, which falls into the framewor…
University of Michigan（密歇根大学）；MITRE（MITRE公司）；Ross School of Business（罗斯商学院）
We develop a notion of projections between sets of probability measures using the geometric properties of the 2-Wasserstein space. It is designed for general multivariate probability measures, is computationally efficient to implement, and provides a unique solution in regular settings. The idea is to work on regular tangent cones of the Wasserstein space using generalized geodesics. Its structure and computational properties make the method applicable in a variety of settings, from causal inference to the analysis of object data. An application to estimating causal effects yields a generaliza…
Wasserstein projection distance for fairness testing of regression models
Department of Computer Science, University of British Columbia（计算机科学系，不列颠哥伦比亚大学）；Department of Pathology and Laboratory Medicine, University of British Columbia（病理学与实验室医学系，不列颠哥伦比亚大学）；Department of Statistics, University of British Columbia（统计学系，不列颠哥伦比亚大学）；BC Cancer Research, Part of Provincial Health Care Authority（BC癌症研究，省卫生保健机构的一部分）；Department of Mathematics, University of British Columbia（数学系，不列颠哥伦比亚大学）
Many variants of the Wasserstein distance have been introduced to reduce its original computational burden. In particular the Sliced-Wasserstein distance (SW), which leverages one-dimensional projections for which a closed-form solution of the Wasserstein distance is available, has received a lot of interest. Yet, it is restricted to data living in Euclidean spaces, while the Wasserstein distance has been studied and used recently on manifolds. We focus more specifically on the sphere, for which we define a novel SW discrepancy, which we call spherical Sliced-Wasserstein, making a first step t…
Wasserstein multivariate auto-regressive models for modeling distributional time series
Université Grenoble Alpes, CNRS, Inria, Grenoble INP, LJK（格勒诺布尔阿尔卑斯大学、国家科学研究中心、法国国家信息与自动化研究所、格勒诺布尔INP、LJK研究中心）；Institut de Mathématiques de Bordeaux, Université de Bordeaux（波尔多数学研究所、波尔多大学）
Simulation of Multivariate Extremes: a Wasserstein-Aitchison GAN approach
Institute of Statistics, Biostatistics and Actuarial Sciences, LIDAM, UCLouvain（统计、生物统计与精算科学研究所，LIDAM，UCLouvain）；CREST, ENSAE Paris（ENSAE巴黎CREST）；Department of Mathematical Sciences, Chalmers University of Technology and University of Gothenburg（数学科学系，查尔姆斯理工大学和哥德堡大学）；Department of Mathematics, KU Leuven（数学系，KU莱顿）
We consider applications involving a large set of instances of projecting points to polytopes. We develop an intuition guided by theoretical and empirical analysis to show that when these instances follow certain structures, a large majority of the projections lie on vertices of the polytopes. To do these projections efficiently we derive a vertex-oriented incremental algorithm to project a point onto any arbitrary polytope, as well as give specific algorithms to cater to simplex projection and polytopes where the unit box is cut by planes. Such settings are especially useful in web-scale appl…
