Stochastic Gradient Descent for Barycenters in Wasserstein Space
Julio Backhoff-Veraguas, Joaquin Fontbona, Gonzalo Rios, Felipe Tobar
备注：
We changed the title, from 'Stochastic Gradient Descent in Wasserstein Space' to 'Stochastic Gradient Descent for Barycenters in Wasserstein Space'. We expanded the literature review, included a new in-depth analysis of convergence rates, and added an appendix containing partial results for the case of possibly non-abs. continuous measures
We present and study a novel algorithm for the computation of 2-Wasserstein population barycenters of absolutely continuous probability measures on Euclidean space. The proposed method can be seen as a stochastic gradient descent procedure in the 2-Wasserstein space, as well as a manifestation of a Law of Large Numbers therein. The algorithm aims at finding a Karcher mean or critical point in this setting, and can be implemented ``online", sequentially using i.i.d. random measures sampled from the population law. We provide natural sufficient conditions for this algorithm to a.s. converge in the Wasserstein space towards the population barycenter, and we introduce a novel, general condition which ensures uniqueness of Karcher means and moreover allows us to obtain explicit, parametric convergence rates for the expected optimality gap. We furthermore study the mini-batch version of this algorithm, and discuss examples of families of population laws to which our method and results can be applied. This work expands and deepens ideas and results introduced in an early version of \cite{backhoff2018bayesian}, in which a statistical application (and numerical implementation) of this method is developed in the context of Bayesian learning.
In this paper, we focus on computational aspects of the Wasserstein barycenter problem. We propose two algorithms to compute Wasserstein barycenters of $m$ discrete measures of size $n$ with accuracy $\e$. The first algorithm, based on mirror prox with a specific norm, meets the complexity of celebrated accelerated iterative Bregman projections (IBP), namely $\widetilde O(mn^2\sqrt n/\e)$, however, with no limitations in contrast to the (accelerated) IBP, which is numerically unstable under small regularization parameter. The second algorithm, based on area-convexity and dual extrapolation, im…
Wasserstein barycenters correspond to optimal solutions of transportation problems for several marginals, and as such have a wide range of applications ranging from economics to statistics and computer science. When the marginal probability measures are absolutely continuous (or vanish on small sets) the theory of Wasserstein barycenters is well-developed (see the seminal paper [1]). However, exact continuous computation of Wasserstein barycenters in this setting is tractable in only a small number of specialized cases. Moreover, in many applications data is given as a set of probability measu…
Wasserstein barycenter is the centroid of a collection of discrete probability distributions which minimizes the average of the $\ell_2$-Wasserstein distance. This paper focuses on the computation of Wasserstein barycenters under the case where the support points are free, which is known to be a severe bottleneck in the D2-clustering due to the large-scale and nonconvexity. We develop an inexact proximal alternating minimization (iPAM) method for computing an approximate Wasserstein barycenter, and provide its global convergence analysis. This method can achieve a good accuracy with a reduced …
We study the problem of the decentralized computation of entropy-regularized semi-discrete Wasserstein barycenters over a network. Building upon recent primal-dual approaches, we propose a sampling gradient quantization scheme that allows efficient communication and computation of approximate barycenters where the factor distributions are stored distributedly on arbitrary networks. The communication and algorithmic complexity of the proposed algorithm are shown, with explicit dependency on the size of the support, the number of distributions, and the desired accuracy. Numerical results validat…
In this thesis, we consider the Wasserstein barycenter problem of discrete probability measures from computational and statistical sides. The statistical focus is estimating the sample size of measures necessary to calculate an approximation for Fréchet mean (barycenter) of a probability distribution with a given precision. For empirical risk minimization approaches, the question of the regularization is also studied together with proposing a new regularization which contributes to the better complexity bounds in comparison with quadratic regularization. The computational focus is developing a…
We present a stochastic algorithm to compute the barycenter of a set of probability distributions under the Wasserstein metric from optimal transport. Unlike previous approaches, our method extends to continuous input distributions and allows the support of the barycenter to be adjusted in each iteration. We tackle the problem without regularization, allowing us to recover a sharp output whose support is contained within the support of the true barycenter. We give examples where our algorithm recovers a more meaningful barycenter than previous work. Our method is versatile and can be extended …
We revisit the problem of recovering a low-rank positive semidefinite matrix from rank-one projections using tools from optimal transport. More specifically, we show that a variational formulation of this problem is equivalent to computing a Wasserstein barycenter. In turn, this new perspective enables the development of new geometric first-order methods with strong convergence guarantees in Bures-Wasserstein distance. Experiments on simulated data demonstrate the advantages of our new methodology over existing methods.
INRIA（法国国家信息与自动化研究所）；Ecole Normale Supérieure（高等师范学校）；PSL Research University（巴黎文理研究大学）；Kayrros SAS（凯罗斯公司）；CNRS（法国国家科学研究中心）
Hyperspectral satellite images report greenhouse gas concentrations worldwide on a daily basis. While taking simple averages of these images over time produces a rough estimate of relative emission rates, atmospheric transport means that simple averages fail to pinpoint the source of these emissions. We propose using Wasserstein barycenters coupled with weather data to average gas concentration data sets and better concentrate the mass around significant sources.
Closing the Gap: Efficient Algorithms for Discrete Wasserstein Barycenters
H. Milton Stewart School of Industrial and Systems Engineering, Georgia Institute of Technology（佐治亚理工学院工业与系统工程学院）；Department of Data Science, City University of Hong Kong（香港城市大学数据科学系）
We study the decentralized distributed computation of discrete approximations for the regularized Wasserstein barycenter of a finite set of continuous probability measures distributedly stored over a network. We assume there is a network of agents/machines/computers, and each agent holds a private continuous probability measure and seeks to compute the barycenter of all the measures in the network by getting samples from its local measure and exchanging information with its neighbors. Motivated by this problem, we develop, and analyze, a novel accelerated primal-dual stochastic gradient method…
Computing Wasserstein barycenters (a.k.a. Optimal Transport barycenters) is a fundamental problem in geometry which has recently attracted considerable attention due to many applications in data science. While there exist polynomial-time algorithms in any fixed dimension, all known running times suffer exponentially in the dimension. It is an open question whether this exponential dependence is improvable to a polynomial dependence. This paper proves that unless P=NP, the answer is no. This uncovers a "curse of dimensionality" for Wasserstein barycenter computation which does not occur for Opt…
In this paper, we focus on the analysis of the regularized Wasserstein barycenter problem. We provide uniqueness and a characterization of the barycenter for two important classes of probability measures: (i) Gaussian distributions and (ii) $q$-Gaussian distributions; each regularized by a particular entropy functional. We propose an algorithm based on gradient projection method in the space of matrices in order to compute these regularized barycenters. We also consider a general class of $φ$-exponential measures, for which only the non-regularized barycenter is studied. Finally, we numericall…
In this paper, we establish a Kantorovich duality for weak optimal total variation transport problems. As consequences, we recover a version of duality formula for partial optimal transports established by Caffarelli and McCann; and we also get another proof of Kantorovich-Rubinstein Theorem for generalized Wasserstein distance $\widetilde{W}_1^{a,b}$ proved before by Piccoli and Rossi. Then we apply our duality formula to study generalized Wasserstein barycenters. We show the existence of these barycenters for measures with compact supports. Finally, we prove the consistency of our barycenter…
Georgia Institute of Technology（佐治亚理工学院）；University of California, Irvine（加利福尼亚大学尔湾分校）
Wasserstein Barycenter is a principled approach to represent the weighted mean of a given set of probability distributions, utilizing the geometry induced by optimal transport. In this work, we present a novel scalable algorithm to approximate the Wasserstein Barycenters aiming at high-dimensional applications in machine learning. Our proposed algorithm is based on the Kantorovich dual formulation of the Wasserstein-2 distance as well as a recent neural network architecture, input convex neural network, that is known to parametrize convex functions. The distinguishing features of our method ar…
We introduce a weak notion of barycenter of a probability measure $μ$ on a metric measure space $(X, d, {\bf m})$, with the metric $d$ and reference measure ${\bf m}$. Under the assumption that optimal transport plans are given by mappings, we prove that our barycenter $B(μ)$ is well defined; it is a probability measure on $X$ supported on the set of the usual metric barycenter points of the given measure $μ$. The definition uses the canonical embedding of the metric space $X$ into its Wasserstein space $P(X)$, pushing a given measure $μ$ forward to a measure on $P(X)$. We then regularize the …
We propose a new \cu{class-optimal} algorithm for the distributed computation of Wasserstein Barycenters over networks. Assuming that each node in a graph has a probability distribution, we prove that every node can reach the barycenter of all distributions held in the network by using local interactions compliant with the topology of the graph. We provide an estimate for the minimum number of communication rounds required for the proposed method to achieve arbitrary relative precision both in the optimality of the solution and the consensus among all agents for undirected fixed networks.
Toland duality and particle approximations for signed Wasserstein barycenters
Université Paris-Saclay, CNRS, Inria, Laboratoire de mathématiques d’Orsay, ParMA（巴黎萨克雷大学、法国国家科学研究中心、Inria、奥赛数学实验室、ParMA）；Université Claude Bernard Lyon 1, ICJ UMR5208, CNRS, Ecole Centrale de Lyon, INSA Lyon, Université Jean Monnet（里昂第一大学、ICJ UMR5208、法国国家科学研究中心、里昂中央理工学院、里昂国立应用科学学院、让·穆兰大学）
Laboratory for Information and Decision Systems (LIDS), Massachusetts Institute of Technology（麻省理工学院信息与决策系统实验室）
Computing Wasserstein barycenters is a fundamental geometric problem with widespread applications in machine learning, statistics, and computer graphics. However, it is unknown whether Wasserstein barycenters can be computed in polynomial time, either exactly or to high precision (i.e., with $\textrm{polylog}(1/\varepsilon)$ runtime dependence). This paper answers these questions in the affirmative for any fixed dimension. Our approach is to solve an exponential-size linear programming formulation by efficiently implementing the corresponding separation oracle using techniques from computation…
Wasserstein barycenters define averages of probability measures in a geometrically meaningful way. Their use is increasingly popular in applied fields, such as image, geometry or language processing. In these fields however, the probability measures of interest are often not accessible in their entirety and the practitioner may have to deal with statistical or computational approximations instead. In this article, we quantify the effect of such approximations on the corresponding barycenters. We show that Wasserstein barycenters depend in a H{ö}lder-continuous way on their marginals under rela…
We consider the problem of computing a Wasserstein barycenter for a set of discrete probability distributions with finite supports, which finds many applications in areas such as statistics, machine learning and image processing. When the support points of the barycenter are pre-specified, this problem can be modeled as a linear programming (LP) problem whose size can be extremely large. To handle this large-scale LP, we analyse the structure of its dual problem, which is conceivably more tractable and can be reformulated as a well-structured convex problem with 3 kinds of block variables and …
This paper discusses the efficiency of Hybrid Primal-Dual (HPD) type algorithms to approximate solve discrete Optimal Transport (OT) and Wasserstein Barycenter (WB) problems, with and without entropic regularization. Our first contribution is an analysis showing that these methods yield state-of-the-art convergence rates, both theoretically and practically. Next, we extend the HPD algorithm with linesearch proposed by Malitsky and Pock in 2018 to the setting where the dual space has a Bregman divergence, and the dual function is relatively strongly convex to the Bregman's kernel. This extensio…
Computing the Wasserstein barycenter of a set of probability measures under the optimal transport metric can quickly become prohibitive for traditional second-order algorithms, such as interior-point methods, as the support size of the measures increases. In this paper, we overcome the difficulty by developing a new adapted interior-point method that fully exploits the problem's special matrix structure to reduce the iteration complexity and speed up the Newton procedure. Different from regularization approaches, our method achieves a well-balanced tradeoff between accuracy and speed. A numeri…
We introduce and study a variant of the Wasserstein distance on the space of probability measures, specially designed to deal with measures whose support has a dendritic, or treelike structure with a particular direction of orientation. Our motivation is the comparison of and interpolation between plants' root systems. We characterize barycenters with respect to this metric, and establish that the interpolations of root-like measures, using this new metric, are also root like, in a certain sense; this property fails for conventional Wasserstein barycenters. We also establish geodesic convexity…
Discrete Wasserstein barycenters correspond to optimal solutions of transportation problems for a set of probability measures with finite support. Discrete barycenters are measures with finite support themselves and exhibit two favorable properties: there always exists one with a provably sparse support, and any optimal transport to the input measures is non-mass splitting. It is open whether a discrete barycenter can be computed in polynomial time. It is possible to find an exact barycenter through linear programming, but these programs may scale exponentially. In this paper, we prove that th…
Efficiently aggregating data from different sources is a challenging problem, particularly when samples from each source are distributed differently. These differences can be inherent to the inference task or present for other reasons: sensors in a sensor network may be placed far apart, affecting their individual measurements. Conversely, it is computationally advantageous to split Bayesian inference tasks across subsets of data, but data need not be identically distributed across subsets. One principled way to fuse probability distributions is via the lens of optimal transport: the Wasserste…
In many applications in statistics and machine learning, the availability of data samples from multiple possibly heterogeneous sources has become increasingly prevalent. On the other hand, in distributionally robust optimization, we seek data-driven decisions which perform well under the most adverse distribution from a nominal distribution constructed from data samples within a certain discrepancy of probability distributions. However, it remains unclear how to achieve such distributional robustness in model learning and estimation when data samples from multiple sources are available. In thi…
In this paper, we propose and analyze an efficient Halpern-Peaceman-Rachford (HPR) algorithm for solving the Wasserstein barycenter problem (WBP) with fixed supports. While the Peaceman-Rachford (PR) splitting method itself may not be convergent for solving the WBP, the HPR algorithm can achieve an $O(1/\varepsilon)$ non-ergodic iteration complexity with respect to the Karush-Kuhn-Tucker (KKT) residual. More interestingly, we propose an efficient procedure with linear time computational complexity to solve the linear systems involved in the subproblems of the HPR algorithm. As a consequence, t…
The discrete Wasserstein barycenter problem is a minimum-cost mass transport problem for a set of probability measures with finite support. In this paper, we show that finding a barycenter of sparse support is hard, even in dimension 2 and for only 3 measures. We prove this claim by showing that a special case of an intimately related decision problem SCMP -- does there exist a measure with a non-mass-splitting transport cost and support size below prescribed bounds? -- is NP-hard for all rational data. Our proof is based on a reduction from planar 3-dimensional matching and follows a strategy…
Weierstrass Institute for Applied Analysis and Stochastics（魏尔斯特拉斯应用分析与随机研究所）；Moscow Institute of Physics and Technology（莫斯科物理技术学院）；Institute for Information Transmission Problems（信息传输问题研究所）
In the machine learning and optimization community, there are two main approaches for the convex risk minimization problem, namely, the Stochastic Approximation (SA) and the Sample Average Approximation (SAA). In terms of oracle complexity (required number of stochastic gradient evaluations), both approaches are considered equivalent on average (up to a logarithmic factor). The total complexity depends on the specific problem, however, starting from work \cite{nemirovski2009robust} it was generally accepted that the SA is better than the SAA. % Nevertheless, in case of large-scale problems SA …
The Hellinger-Kantorovich (HK) distance is an unbalanced extension of the Wasserstein-2 distance. It was shown recently that the HK barycenter exhibits a much more complex behaviour than the Wasserstein barycenter. Motivated by this observation we study the HK barycenter in more detail for the case where the input measures are an uncountable collection of Dirac measures, in particular the dependency on the length scale parameter of HK, the question whether the HK barycenter is discrete or continuous and the relation between the expected and the empirical barycenter. The analytical results are …
HSE University（俄罗斯高等经济大学）；Moscow Institute of Physics and Technology（莫斯科物理技术学院）；Institute for Information Transmission Problems（信息传输问题研究所）；Weierstrass Institute for Applied Analysis and Stochastics（魏尔斯特拉斯应用分析与随机研究所）
We consider the population Wasserstein barycenter problem for random probability measures supported on a finite set of points and generated by an online stream of data. This leads to a complicated stochastic optimization problem where the objective is given as an expectation of a function given as a solution to a random optimization problem. We employ the structure of the problem and obtain a convex-concave stochastic saddle-point reformulation of this problem. In the setting when the distribution of random probability measures is discrete, we propose a stochastic optimization algorithm and es…
Wasserstein Barycenter Gaussian Process based Bayesian Optimization
Department of Economics Management and Statistics, University of Milano-Bicocca（经济管理统计系，米兰-比科卡大学）；Department of Computer Science Systems and Communication, University of Milano-Bicocca（计算机科学系统与通信系，米兰-比科卡大学）
We study the complexity of approximating Wassertein barycenter of $m$ discrete measures, or histograms of size $n$ by contrasting two alternative approaches, both using entropic regularization. The first approach is based on the Iterative Bregman Projections (IBP) algorithm for which our novel analysis gives a complexity bound proportional to $\frac{mn^2}{\varepsilon^2}$ to approximate the original non-regularized barycenter. Using an alternative accelerated-gradient-descent-based approach, we obtain a complexity proportional to $\frac{mn^{2.5}}{\varepsilon} $. As a byproduct, we show that the…
