Optimal transport (\OT) theory defines a powerful set of tools to compare probability distributions. \OT~suffers however from a few drawbacks, computational and statistical, which have encouraged the proposal of several regularized variants of OT in the recent literature, one of the most notable being the \textit{sliced} formulation, which exploits the closed-form formula between univariate distributions by projecting high-dimensional measures onto random lines. We consider in this work a more general family of ground metrics, namely \textit{tree metrics}, which also yield fast closed-form computations and negative definite, and of which the sliced-Wasserstein distance is a particular case (the tree is a chain). We propose the tree-sliced Wasserstein distance, computed by averaging the Wasserstein distance between these measures using random tree metrics, built adaptively in either low or high-dimensional spaces. Exploiting the negative definiteness of that distance, we also propose a positive definite kernel, and test it against other baselines on a few benchmark tasks.
Sliced Wasserstein distances preserve properties of classic Wasserstein distances while being more scalable for computation and estimation in high dimensions. The goal of this work is to quantify this scalability from three key aspects: (i) empirical convergence rates; (ii) robustness to data contamination; and (iii) efficient computational methods. For empirical convergence, we derive fast rates with explicit dependence of constants on dimension, subject to log-concavity of the population distributions. For robustness, we characterize minimax optimal, dimension-free robust estimation risks, a…
Optimal transport distances, otherwise known as Wasserstein distances, have recently drawn ample attention in computer vision and machine learning as a powerful discrepancy measure for probability distributions. The recent developments on alternative formulations of the optimal transport have allowed for faster solutions to the problem and has revamped its practical applications in machine learning. In this paper, we exploit the widely used kernel methods and provide a family of provably positive definite kernels based on the Sliced Wasserstein distance and demonstrate the benefits of these ke…
University College London（伦敦大学学院）；Facebook AI Research（脸书人工智能研究院）；University of Cambridge（剑桥大学）；University of California, Berkeley（加州大学伯克利分校）；Imperial College London（帝国理工学院）
Multi-marginal optimal transport enables one to compare multiple probability measures, which increasingly finds application in multi-task learning problems. One practical limitation of multi-marginal transport is computational scalability in the number of measures, samples and dimensionality. In this work, we propose a multi-marginal optimal transport paradigm based on random one-dimensional projections, whose (generalized) distance we term the sliced multi-marginal Wasserstein distance. To construct this distance, we introduce a characterization of the one-dimensional multi-marginal Kantorovi…
Variational Inference approximates an unnormalized distribution via the minimization of Kullback-Leibler (KL) divergence. Although this divergence is efficient for computation and has been widely used in applications, it suffers from some unreasonable properties. For example, it is not a proper metric, i.e., it is non-symmetric and does not preserve the triangle inequality. On the other hand, optimal transport distances recently have shown some advantages over KL divergence. With the help of these advantages, we propose a new variational inference method by minimizing sliced Wasserstein distan…
Recently used in various machine learning contexts, the Gromov-Wasserstein distance (GW) allows for comparing distributions whose supports do not necessarily lie in the same metric space. However, this Optimal Transport (OT) distance requires solving a complex non convex quadratic program which is most of the time very costly both in time and memory. Contrary to GW, the Wasserstein distance (W) enjoys several properties (e.g. duality) that permit large scale optimization. Among those, the solution of W on the real line, that only requires sorting discrete samples in 1D, allows defining the Sli…
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）
Sliced Wasserstein (SW) distance has been widely used in different application scenarios since it can be scaled to a large number of supports without suffering from the curse of dimensionality. The value of sliced Wasserstein distance is the average of transportation cost between one-dimensional representations (projections) of original measures that are obtained by Radon Transform (RT). Despite its efficiency in the number of supports, estimating the sliced Wasserstein requires a relatively large number of projections in high-dimensional settings. Therefore, for applications where the number …
Télécom Paris（巴黎电信学院）；Institut Polytechnique de Paris（巴黎理工学院）；ENS Cachan（卡尚高等师范学校）；CNRS（法国国家科学研究中心）；Université Paris-Saclay（巴黎萨克雷大学）；University of Oxford（牛津大学）
Minimum expected distance estimation (MEDE) algorithms have been widely used for probabilistic models with intractable likelihood functions and they have become increasingly popular due to their use in implicit generative modeling (e.g. Wasserstein generative adversarial networks, Wasserstein autoencoders). Emerging from computational optimal transport, the Sliced-Wasserstein (SW) distance has become a popular choice in MEDE thanks to its simplicity and computational benefits. While several studies have reported empirical success on generative modeling with SW, the theoretical properties of su…
HRL Laboratories, LLC（HRL实验室有限责任公司）；University of Virginia（弗吉尼亚大学）
Gaussian mixture models (GMM) are powerful parametric tools with many applications in machine learning and computer vision. Expectation maximization (EM) is the most popular algorithm for estimating the GMM parameters. However, EM guarantees only convergence to a stationary point of the log-likelihood function, which could be arbitrarily worse than the optimal solution. Inspired by the relationship between the negative log-likelihood function and the Kullback-Leibler (KL) divergence, we propose an alternative formulation for estimating the GMM parameters using the sliced Wasserstein distance, …
Towards Understanding Gradient Dynamics of the Sliced-Wasserstein Distance via Critical Point Analysis
Laboratoire de Mathématiques d'Orsay, Université Paris-Saclay, Gif-sur-Yvette, France（巴黎-萨克雷大学奥赛数学实验室）；Centre de recherche en économie et statistique, ENSAE, Palaiseau, France（经济与统计研究中心）
University College London（伦敦大学学院）；Italian Institute of Technology（意大利理工学院）
The problem of learning functions over spaces of probabilities - or distribution regression - is gaining significant interest in the machine learning community. A key challenge behind this problem is to identify a suitable representation capturing all relevant properties of the underlying functional mapping. A principled approach to distribution regression is provided by kernel mean embeddings, which lifts kernel-induced similarity on the input domain at the probability level. This strategy effectively tackles the two-stage sampling nature of the problem, enabling one to derive estimators with…
Kyoto University（京都大学）；RIKEN AIP（理化学研究所先进智能项目中心）；Facebook AI Research（脸书人工智能研究院）；SliceX AI（SliceX人工智能）
The Wasserstein barycenter has been widely studied in various fields, including natural language processing, and computer vision. However, it requires a high computational cost to solve the Wasserstein barycenter problem because the computation of the Wasserstein distance requires a quadratic time with respect to the number of supports. By contrast, the Wasserstein distance on a tree, called the tree-Wasserstein distance, can be computed in linear time and allows for the fast comparison of a large number of distributions. In this study, we propose a barycenter under the tree-Wasserstein distan…
Inria（法国国家信息与自动化研究所）；LTCI, Télécom ParisTech, Université Paris-Saclay（巴黎-萨克雷大学巴黎高等电信学院信息处理与通信实验室）；Institute of Mathematics, Polish Academy of Sciences（波兰科学院数学研究所）
By building upon the recent theory that established the connection between implicit generative modeling (IGM) and optimal transport, in this study, we propose a novel parameter-free algorithm for learning the underlying distributions of complicated datasets and sampling from them. The proposed algorithm is based on a functional optimization problem, which aims at finding a measure that is close to the data distribution as much as possible and also expressive enough for generative modeling purposes. We formulate the problem as a gradient flow in the space of probability measures. The connection…
Statistical and Computational Guarantees of Kernel Max-Sliced Wasserstein Distances
School of Data Science, The Chinese University of Hong Kong, Shenzhen, Shenzhen, China（数据科学学院，香港中文大学（深圳））；School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen, Shenzhen, China（人工智能学院，香港中文大学（深圳））；Department of Mathematics, Michigan State University, East Lansing, USA（数学系，密歇根州立大学）；School of Industrial and Systems Engineering, Georgia Institute of Technology, Atlanta, USA（工业与系统工程学院，佐治亚理工学院）
Sliced Wasserstein Steering between Gaussian Measures
Department of Information Physics and Computing, The University of Tokyo（信息物理与计算系，东京大学）；Department of Decision and Control Systems and Department of Mathematics, KTH Royal Institute of Technology（决策与控制系统系和数学系，皇家理工学院）
Vertical Sliced Wasserstein Distances for Meta-Measures
College of Connected Computing, Vanderbilt University（范德堡大学连接计算学院）；Toyota Motor North America, InfoTech Labs（北美丰田汽车公司信息技术实验室）；University of California, Davis（加州大学戴维斯分校）；Purdue University（普渡大学）；Florida State University（佛罗里达州立大学）
National University of Singapore（新加坡国立大学）；FPT Software AI Center（FPT软件AI中心）；Movian AI；The Institute of Statistical Mathematics（统计数学研究所）；RIKEN AIP（理化学研究所革新智能创造领域）
Duke University（杜克大学）；Infinia ML Inc.（Infinia ML公司）
Traditional multi-view learning methods often rely on two assumptions: ($i$) the samples in different views are well-aligned, and ($ii$) their representations in latent space obey the same distribution. Unfortunately, these two assumptions may be questionable in practice, which limits the application of multi-view learning. In this work, we propose a hierarchical optimal transport (HOT) method to mitigate the dependency on these two assumptions. Given unaligned multi-view data, the HOT method penalizes the sliced Wasserstein distance between the distributions of different views. These sliced W…
Criteo AI Lab（Criteo 人工智能实验室）；Université Technologique de Compiègne（贡比涅技术大学）；Université de Rouen（鲁昂大学）；INSA de Rouen（鲁昂国立应用科学学院）
Gaussian smoothed sliced Wasserstein distance has been recently introduced for comparing probability distributions, while preserving privacy on the data. It has been shown, in applications such as domain adaptation, to provide performances similar to its non-private (non-smoothed) counterpart. However, the computational and statistical properties of such a metric is not yet been well-established. In this paper, we analyze the theoretical properties of this distance as well as those of generalized versions denoted as Gaussian smoothed sliced divergences. We show that smoothing and slicing prese…
Université de Technologie de Compiègne（贡比涅技术大学）；INSA-Rouen（鲁昂国立应用科学学院）；Université de Rouen Normandie（鲁昂诺曼底大学）；Criteo AI Lab（Criteo人工智能实验室）
Optimal Transport (OT) metrics allow for defining discrepancies between two probability measures. Wasserstein distance is for longer the celebrated OT-distance frequently-used in the literature, which seeks probability distributions to be supported on the $\textit{same}$ metric space. Because of its high computational complexity, several approximate Wasserstein distances have been proposed based on entropy regularization or on slicing, and one-dimensional Wassserstein computation. In this paper, we propose a novel extension of Wasserstein distance to compare two incomparable distributions, tha…
École Polytechnique（巴黎综合理工学院）；University of Texas, Austin（德克萨斯大学奥斯汀分校）；Hanoi University of Science and Technology（河内科技大学）
Generalized sliced Wasserstein distance is a variant of sliced Wasserstein distance that exploits the power of non-linear projection through a given defining function to better capture the complex structures of the probability distributions. Similar to sliced Wasserstein distance, generalized sliced Wasserstein is defined as an expectation over random projections which can be approximated by the Monte Carlo method. However, the complexity of that approximation can be expensive in high-dimensional settings. To that end, we propose to form deterministic and fast approximations of the generalized…
HRL Laboratories, LLC（HRL实验室有限责任公司）；University of Virginia（弗吉尼亚大学）
In this paper we study generative modeling via autoencoders while using the elegant geometric properties of the optimal transport (OT) problem and the Wasserstein distances. We introduce Sliced-Wasserstein Autoencoders (SWAE), which are generative models that enable one to shape the distribution of the latent space into any samplable probability distribution without the need for training an adversarial network or defining a closed-form for the distribution. In short, we regularize the autoencoder loss with the sliced-Wasserstein distance between the distribution of the encoded training samples…
Seeking informative projecting directions has been an important task in utilizing sliced Wasserstein distance in applications. However, finding these directions usually requires an iterative optimization procedure over the space of projecting directions, which is computationally expensive. Moreover, the computational issue is even more severe in deep learning applications, where computing the distance between two mini-batch probability measures is repeated several times. This nested loop has been one of the main challenges that prevent the usage of sliced Wasserstein distances based on good pr…
Leveraging Optimal Transport via Projections on Subspaces for Machine Learning Applications
Université de Bretagne Sud（南布列塔尼大学）；Laboratoire de Mathématiques de Bretagne Atlantique（大西洋布列塔尼数学实验室）；CNRS（法国国家科学研究中心）；École Normale Supérieure（巴黎高等师范学院）；Technische Universität Berlin（柏林工业大学）；Université Paris Cité（巴黎西岱大学）；Sony Computer Science Laboratories Inc（索尼计算机科学实验室公司）；Harvard SEAS（哈佛大学工程与应用科学学院）；IMT Atlantique（大西洋国立高等工业技术与矿业学校）
Many variants of the Wasserstein distance have been introduced to reduce its original computational burden. In particular the Sliced-Wasserstein distance (SW), which leverages one-dimensional projections for which a closed-form solution of the Wasserstein distance is available, has received a lot of interest. Yet, it is restricted to data living in Euclidean spaces, while the Wasserstein distance has been studied and used recently on manifolds. We focus more specifically on the sphere, for which we define a novel SW discrepancy, which we call spherical Sliced-Wasserstein, making a first step t…
HRL Laboratories, LLC.（HRL实验室有限责任公司）；Télécom ParisTech（巴黎高等电信学院）；University of Virginia（弗吉尼亚大学）
The Wasserstein distance and its variations, e.g., the sliced-Wasserstein (SW) distance, have recently drawn attention from the machine learning community. The SW distance, specifically, was shown to have similar properties to the Wasserstein distance, while being much simpler to compute, and is therefore used in various applications including generative modeling and general supervised/unsupervised learning. In this paper, we first clarify the mathematical connection between the SW distance and the Radon transform. We then utilize the generalized Radon transform to define a new family of dista…
Criteo AI Lab（Criteo人工智能实验室）；Université de Rouen Normandie（鲁昂诺曼底大学）
Developing machine learning methods that are privacy preserving is today a central topic of research, with huge practical impacts. Among the numerous ways to address privacy-preserving learning, we here take the perspective of computing the divergences between distributions under the Differential Privacy (DP) framework -- being able to compute divergences between distributions is pivotal for many machine learning problems, such as learning generative models or domain adaptation problems. Instead of resorting to the popular gradient-based sanitization method for DP, we tackle the problem at its…
VinAI Research（VinAI研究院）；University of Texas, Austin（德克萨斯大学奥斯汀分校）
Relational regularized autoencoder (RAE) is a framework to learn the distribution of data by minimizing a reconstruction loss together with a relational regularization on the latent space. A recent attempt to reduce the inner discrepancy between the prior and aggregated posterior distributions is to incorporate sliced fused Gromov-Wasserstein (SFG) between these distributions. That approach has a weakness since it treats every slicing direction similarly, meanwhile several directions are not useful for the discriminative task. To improve the discrepancy and consequently the relational regulari…
University of Surrey（萨里大学）；University of Edinburgh（爱丁堡大学）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）
While theoretically appealing, the application of the Wasserstein distance to large-scale machine learning problems has been hampered by its prohibitive computational cost. The sliced Wasserstein distance and its variants improve the computational efficiency through the random projection, yet they suffer from low accuracy if the number of projections is not sufficiently large, because the majority of projections result in trivially small values. In this work, we propose a new family of distance metrics, called augmented sliced Wasserstein distances (ASWDs), constructed by first mapping samples…
The conventional sliced Wasserstein is defined between two probability measures that have realizations as vectors. When comparing two probability measures over images, practitioners first need to vectorize images and then project them to one-dimensional space by using matrix multiplication between the sample matrix and the projection matrix. After that, the sliced Wasserstein is evaluated by averaging the two corresponding one-dimensional projected probability measures. However, this approach has two limitations. The first limitation is that the spatial structure of images is not captured effi…
VinAI Research（VinAI研究院）；University of Texas, Austin（得克萨斯大学奥斯汀分校）
Sliced-Wasserstein distance (SW) and its variant, Max Sliced-Wasserstein distance (Max-SW), have been used widely in the recent years due to their fast computation and scalability even when the probability measures lie in a very high dimensional space. However, SW requires many unnecessary projection samples to approximate its value while Max-SW only uses the most important projection, which ignores the information of other useful directions. In order to account for these weaknesses, we propose a novel distance, named Distributional Sliced-Wasserstein distance (DSW), that finds an optimal dist…
LTCI, Télécom Paris, Institut Polytechnique de Paris（巴黎电信学院，巴黎综合理工学院）；ESSEC Business School（ESSEC商学院）
The Sliced-Wasserstein distance (SW) is being increasingly used in machine learning applications as an alternative to the Wasserstein distance and offers significant computational and statistical benefits. Since it is defined as an expectation over random projections, SW is commonly approximated by Monte Carlo. We adopt a new perspective to approximate SW by making use of the concentration of measure phenomenon: under mild assumptions, one-dimensional projections of a high-dimensional random vector are approximately Gaussian. Based on this observation, we develop a simple deterministic approxi…
In this work, we connect two distinct concepts for unsupervised domain adaptation: feature distribution alignment between domains by utilizing the task-specific decision boundary and the Wasserstein metric. Our proposed sliced Wasserstein discrepancy (SWD) is designed to capture the natural notion of dissimilarity between the outputs of task-specific classifiers. It provides a geometrically meaningful guidance to detect target samples that are far from the support of the source and enables efficient distribution alignment in an end-to-end trainable fashion. In the experiments, we validate the …
