最优传输（OT）在机器学习、数据科学和计算机视觉中已变得极为流行。OT 问题的核心假设是源测度与目标测度具有相等的总质量，这限制了其应用。最优部分传输（OPT）是最近针对这一局限提出的解决方案。与 OT 问题类似，OPT 的计算依赖于求解一个线性规划问题（通常在高维空间中），这可能在计算上变得难以承受。在本文中，我们提出了一种高效算法，用于计算一维空间中两个非负测度之间的 OPT 问题。接着，遵循切片 OT 距离的思想，我们利用切片来定义切片 OPT 距离。最后，我们在各种数值实验中展示了基于切片 OPT 的方法在计算和精度方面的优势。特别地，我们展示了所提出的 Sliced-OPT 在含噪声点云配准中的应用。
英文摘要：
Optimal transport (OT) has become exceedingly popular in machine learning, data science, and computer vision. The core assumption in the OT problem is the equal total amount of mass in source and target measures, which limits its application. Optimal Partial Transport (OPT) is a recently proposed solution to this limitation. Similar to the OT problem, the computation of OPT relies on solving a linear programming problem (often in high dimensions), which can become computationally prohibitive. In this paper, we propose an efficient algorithm for calculating the OPT problem between two non-negative measures in one dimension. Next, following the idea of sliced OT distances, we utilize slicing to define the sliced OPT distance. Finally, we demonstrate the computational and accuracy benefits of the sliced OPT-based method in various numerical experiments. In particular, we show an application of our proposed Sliced-OPT in noisy point cloud registration.
University College London（伦敦大学学院）；Facebook AI Research（脸书人工智能研究院）；University of Cambridge（剑桥大学）；University of California, Berkeley（加州大学伯克利分校）；Imperial College London（帝国理工学院）
Multi-marginal optimal transport enables one to compare multiple probability measures, which increasingly finds application in multi-task learning problems. One practical limitation of multi-marginal transport is computational scalability in the number of measures, samples and dimensionality. In this work, we propose a multi-marginal optimal transport paradigm based on random one-dimensional projections, whose (generalized) distance we term the sliced multi-marginal Wasserstein distance. To construct this distance, we introduce a characterization of the one-dimensional multi-marginal Kantorovi…
Optimal transport distances, otherwise known as Wasserstein distances, have recently drawn ample attention in computer vision and machine learning as a powerful discrepancy measure for probability distributions. The recent developments on alternative formulations of the optimal transport have allowed for faster solutions to the problem and has revamped its practical applications in machine learning. In this paper, we exploit the widely used kernel methods and provide a family of provably positive definite kernels based on the Sliced Wasserstein distance and demonstrate the benefits of these ke…
Variational Inference approximates an unnormalized distribution via the minimization of Kullback-Leibler (KL) divergence. Although this divergence is efficient for computation and has been widely used in applications, it suffers from some unreasonable properties. For example, it is not a proper metric, i.e., it is non-symmetric and does not preserve the triangle inequality. On the other hand, optimal transport distances recently have shown some advantages over KL divergence. With the help of these advantages, we propose a new variational inference method by minimizing sliced Wasserstein distan…
Inria（法国国家信息与自动化研究所）；LTCI, Télécom ParisTech, Université Paris-Saclay（巴黎-萨克雷大学巴黎高等电信学院信息处理与通信实验室）；Institute of Mathematics, Polish Academy of Sciences（波兰科学院数学研究所）
By building upon the recent theory that established the connection between implicit generative modeling (IGM) and optimal transport, in this study, we propose a novel parameter-free algorithm for learning the underlying distributions of complicated datasets and sampling from them. The proposed algorithm is based on a functional optimization problem, which aims at finding a measure that is close to the data distribution as much as possible and also expressive enough for generative modeling purposes. We formulate the problem as a gradient flow in the space of probability measures. The connection…
RIKEN AIP（理化学研究所AIP）；Kyoto University（京都大学）；The Institute of Statistical Mathematics, Japan（日本统计数理研究所）；Google Brain, Paris（谷歌大脑巴黎）；CREST - ENSAE（CREST - 法国国立统计与经济管理学院）
Optimal transport (\OT) theory defines a powerful set of tools to compare probability distributions. \OT~suffers however from a few drawbacks, computational and statistical, which have encouraged the proposal of several regularized variants of OT in the recent literature, one of the most notable being the \textit{sliced} formulation, which exploits the closed-form formula between univariate distributions by projecting high-dimensional measures onto random lines. We consider in this work a more general family of ground metrics, namely \textit{tree metrics}, which also yield fast closed-form com…
Duke University（杜克大学）；Infinia ML Inc.（Infinia ML公司）
Traditional multi-view learning methods often rely on two assumptions: ($i$) the samples in different views are well-aligned, and ($ii$) their representations in latent space obey the same distribution. Unfortunately, these two assumptions may be questionable in practice, which limits the application of multi-view learning. In this work, we propose a hierarchical optimal transport (HOT) method to mitigate the dependency on these two assumptions. Given unaligned multi-view data, the HOT method penalizes the sliced Wasserstein distance between the distributions of different views. These sliced W…
Recently used in various machine learning contexts, the Gromov-Wasserstein distance (GW) allows for comparing distributions whose supports do not necessarily lie in the same metric space. However, this Optimal Transport (OT) distance requires solving a complex non convex quadratic program which is most of the time very costly both in time and memory. Contrary to GW, the Wasserstein distance (W) enjoys several properties (e.g. duality) that permit large scale optimization. Among those, the solution of W on the real line, that only requires sorting discrete samples in 1D, allows defining the Sli…
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）
Sliced Wasserstein (SW) distance has been widely used in different application scenarios since it can be scaled to a large number of supports without suffering from the curse of dimensionality. The value of sliced Wasserstein distance is the average of transportation cost between one-dimensional representations (projections) of original measures that are obtained by Radon Transform (RT). Despite its efficiency in the number of supports, estimating the sliced Wasserstein requires a relatively large number of projections in high-dimensional settings. Therefore, for applications where the number …
Sliced Wasserstein Steering between Gaussian Measures
Department of Information Physics and Computing, The University of Tokyo（信息物理与计算系，东京大学）；Department of Decision and Control Systems and Department of Mathematics, KTH Royal Institute of Technology（决策与控制系统系和数学系，皇家理工学院）
Towards Understanding Gradient Dynamics of the Sliced-Wasserstein Distance via Critical Point Analysis
Laboratoire de Mathématiques d'Orsay, Université Paris-Saclay, Gif-sur-Yvette, France（巴黎-萨克雷大学奥赛数学实验室）；Centre de recherche en économie et statistique, ENSAE, Palaiseau, France（经济与统计研究中心）
University College London（伦敦大学学院）；Italian Institute of Technology（意大利理工学院）
The problem of learning functions over spaces of probabilities - or distribution regression - is gaining significant interest in the machine learning community. A key challenge behind this problem is to identify a suitable representation capturing all relevant properties of the underlying functional mapping. A principled approach to distribution regression is provided by kernel mean embeddings, which lifts kernel-induced similarity on the input domain at the probability level. This strategy effectively tackles the two-stage sampling nature of the problem, enabling one to derive estimators with…
Télécom Paris（巴黎电信学院）；Institut Polytechnique de Paris（巴黎理工学院）；ENS Cachan（卡尚高等师范学校）；CNRS（法国国家科学研究中心）；Université Paris-Saclay（巴黎萨克雷大学）；University of Oxford（牛津大学）
Minimum expected distance estimation (MEDE) algorithms have been widely used for probabilistic models with intractable likelihood functions and they have become increasingly popular due to their use in implicit generative modeling (e.g. Wasserstein generative adversarial networks, Wasserstein autoencoders). Emerging from computational optimal transport, the Sliced-Wasserstein (SW) distance has become a popular choice in MEDE thanks to its simplicity and computational benefits. While several studies have reported empirical success on generative modeling with SW, the theoretical properties of su…
Vertical Sliced Wasserstein Distances for Meta-Measures
College of Connected Computing, Vanderbilt University（范德堡大学连接计算学院）；Toyota Motor North America, InfoTech Labs（北美丰田汽车公司信息技术实验室）；University of California, Davis（加州大学戴维斯分校）；Purdue University（普渡大学）；Florida State University（佛罗里达州立大学）
Leveraging Optimal Transport via Projections on Subspaces for Machine Learning Applications
Université de Bretagne Sud（南布列塔尼大学）；Laboratoire de Mathématiques de Bretagne Atlantique（大西洋布列塔尼数学实验室）；CNRS（法国国家科学研究中心）；École Normale Supérieure（巴黎高等师范学院）；Technische Universität Berlin（柏林工业大学）；Université Paris Cité（巴黎西岱大学）；Sony Computer Science Laboratories Inc（索尼计算机科学实验室公司）；Harvard SEAS（哈佛大学工程与应用科学学院）；IMT Atlantique（大西洋国立高等工业技术与矿业学校）
Statistical and Computational Guarantees of Kernel Max-Sliced Wasserstein Distances
School of Data Science, The Chinese University of Hong Kong, Shenzhen, Shenzhen, China（数据科学学院，香港中文大学（深圳））；School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen, Shenzhen, China（人工智能学院，香港中文大学（深圳））；Department of Mathematics, Michigan State University, East Lansing, USA（数学系，密歇根州立大学）；School of Industrial and Systems Engineering, Georgia Institute of Technology, Atlanta, USA（工业与系统工程学院，佐治亚理工学院）
National University of Singapore（新加坡国立大学）；FPT Software AI Center（FPT软件AI中心）；Movian AI；The Institute of Statistical Mathematics（统计数学研究所）；RIKEN AIP（理化学研究所革新智能创造领域）
Computing optimal transport (OT) between measures in high dimensions is doomed by the curse of dimensionality. A popular approach to avoid this curse is to project input measures on lower-dimensional subspaces (1D lines in the case of sliced Wasserstein distances), solve the OT problem between these reduced measures, and settle for the Wasserstein distance between these reductions, rather than that between the original measures. This approach is however difficult to extend to the case in which one wants to compute an OT map (a Monge map) between the original measures. Since computations are ca…
RIKEN AIP（理化学研究所先进智能项目中心）；The University of Akron（阿克伦大学）
Optimal transport (OT) theory provides powerful tools to compare probability measures. However, OT is limited to nonnegative measures having the same mass, and suffers serious drawbacks about its computation and statistics. This leads to several proposals of regularized variants of OT in the recent literature. In this work, we consider an \textit{entropy partial transport} (EPT) problem for nonnegative measures on a tree having different masses. The EPT is shown to be equivalent to a standard complete OT problem on a one-node extended tree. We derive its dual formulation, then leverage this to…
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；VinAI Research（VinAI研究院）
Mini-batch optimal transport (m-OT) has been widely used recently to deal with the memory issue of OT in large-scale applications. Despite their practicality, m-OT suffers from misspecified mappings, namely, mappings that are optimal on the mini-batch level but are partially wrong in the comparison with the optimal transportation plan between the original measures. Motivated by the misspecified mappings issue, we propose a novel mini-batch method by using partial optimal transport (POT) between mini-batch empirical measures, which we refer to as mini-batch partial optimal transport (m-POT). Le…
ESPFormer: Doubly-Stochastic Attention with Expected Sliced Transport Plans
Department of Computer Science, Vanderbilt University, Nashville, TN, USA.（计算机科学系，范德比尔特大学）；Department of Computer Science, Duke University, Durham, NC, USA.（计算机科学系，杜克大学）；Department of Biostatistics and Bioinformatics, Duke University, Durham, NC, USA.（生物统计学与生物信息学系，杜克大学）
We propose a new colour transfer method with Optimal Transport (OT) to transfer the colour of a sourceimage to match the colour of a target image of the same scene that may exhibit large motion changes betweenimages. By definition OT does not take into account any available information about correspondences whencomputing the optimal solution. To tackle this problem we propose to encode overlapping neighborhoodsof pixels using both their colour and spatial correspondences estimated using motion estimation. We solvethe high dimensional problem in 1D space using an iterative projection approach. …
Classical optimal transport problem seeks a transportation map that preserves the total mass betwenn two probability distributions, requiring their mass to be the same. This may be too restrictive in certain applications such as color or shape matching, since the distributions may have arbitrary masses and/or that only a fraction of the total mass has to be transported. Several algorithms have been devised for computing partial Wasserstein metrics that rely on an entropic regularization, but when it comes with exact solutions, almost no partial formulation of neither Wasserstein nor Gromov-Was…
University of California, Berkeley（加州大学伯克利分校）；CREST - ENSAE（法国国家统计与经济管理学院 Crest 实验室）；Google Brain（谷歌大脑）
Optimal transport (OT) distances are increasingly used as loss functions for statistical inference, notably in the learning of generative models or supervised learning. Yet, the behavior of minimum Wasserstein estimators is poorly understood, notably in high-dimensional regimes or under model misspecification. In this work we adopt the viewpoint of projection robust (PR) OT, which seeks to maximize the OT cost between two measures by choosing a $k$-dimensional subspace onto which they can be projected. Our first contribution is to establish several fundamental statistical properties of PR Wass…
University of Texas, Austin（德克萨斯大学奥斯汀分校）；VinAI Research（VinAI研究院）
We study the multi-marginal partial optimal transport (POT) problem between $m$ discrete (unbalanced) measures with at most $n$ supports. We first prove that we can obtain two equivalence forms of the multimarginal POT problem in terms of the multimarginal optimal transport problem via novel extensions of cost tensor. The first equivalence form is derived under the assumptions that the total masses of each measure are sufficiently close while the second equivalence form does not require any conditions on these masses but at the price of more sophisticated extended cost tensor. Our proof techni…
University of California, Berkeley（加州大学伯克利分校）；Lawrence Berkeley National Laboratory（劳伦斯伯克利国家实验室）
We develop an iterative (greedy) deep learning (DL) algorithm which is able to transform an arbitrary probability distribution function (PDF) into the target PDF. The model is based on iterative Optimal Transport of a series of 1D slices, matching on each slice the marginal PDF to the target. The axes of the orthogonal slices are chosen to maximize the PDF difference using Wasserstein distance at each iteration, which enables the algorithm to scale well to high dimensions. As special cases of this algorithm, we introduce two sliced iterative Normalizing Flow (SINF) models, which map from the d…
An increasing number of machine learning tasks deal with learning representations from set-structured data. Solutions to these problems involve the composition of permutation-equivariant modules (e.g., self-attention, or individual processing via feed-forward neural networks) and permutation-invariant modules (e.g., global average pooling, or pooling by multi-head attention). In this paper, we propose a geometrically-interpretable framework for learning representations from set-structured data, which is rooted in the optimal mass transportation problem. In particular, we treat elements of a se…
