We present a versatile adaptation of existing dimensionality reduction (DR) objectives, enabling the simultaneous reduction of both sample and feature sizes. Correspondances between input and embedding samples are computed through a semi-relaxed Gromov-Wasserstein optimal transport (OT) problem. When the embedding sample size matches that of the input, our model recovers classical popular DR models. When the embedding's dimensionality is unconstrained, we show that the OT plan delivers a competitive hard clustering. We emphasize the importance of intermediate stages that blend DR and clustering for summarizing real data and apply our method to visualize datasets of images.
Duke University（杜克大学）；Infinia ML, Inc.（Infinia ML公司）
We propose a novel fused Gromov-Wasserstein alignment method to jointly learn the Hawkes processes in different event spaces, and align their event types. Given two Hawkes processes, we use fused Gromov-Wasserstein discrepancy to measure their dissimilarity, which considers both the Wasserstein discrepancy based on their base intensities and the Gromov-Wasserstein discrepancy based on their infectivity matrices. Accordingly, the learned optimal transport reflects the correspondence between the event types of these two Hawkes processes. The Hawkes processes and their optimal transport are learn…
Optimal transport theory has recently found many applications in machine learning thanks to its capacity for comparing various machine learning objects considered as distributions. The Kantorovitch formulation, leading to the Wasserstein distance, focuses on the features of the elements of the objects but treat them independently, whereas the Gromov-Wasserstein distance focuses only on the relations between the elements, depicting the structure of the object, yet discarding its features. In this paper we propose to extend these distances in order to encode simultaneously both the feature and s…
Recently used in various machine learning contexts, the Gromov-Wasserstein distance (GW) allows for comparing distributions whose supports do not necessarily lie in the same metric space. However, this Optimal Transport (OT) distance requires solving a complex non convex quadratic program which is most of the time very costly both in time and memory. Contrary to GW, the Wasserstein distance (W) enjoys several properties (e.g. duality) that permit large scale optimization. Among those, the solution of W on the real line, that only requires sorting discrete samples in 1D, allows defining the Sli…
A dimensionality reduction technique based on the Gromov-Wasserstein distance
Instituto Federal de Educação, Ciência e Tecnologia do Ceará（塞阿拉联邦教育、科学与技术研究所）；Federal University of Ceara（塞阿拉联邦大学）；Université Paris-Saclay, CEA, List（巴黎-萨克雷大学，CEA，List）
Infinia ML, Inc.（Infinia ML公司）；Duke University（杜克大学）；Georgia Institute of Technology（佐治亚理工学院）
A novel Gromov-Wasserstein learning framework is proposed to jointly match (align) graphs and learn embedding vectors for the associated graph nodes. Using Gromov-Wasserstein discrepancy, we measure the dissimilarity between two graphs and find their correspondence, according to the learned optimal transport. The node embeddings associated with the two graphs are learned under the guidance of the optimal transport, the distance of which not only reflects the topological structure of each graph but also yields the correspondence across the graphs. These two learning steps are mutually-beneficia…
Infinia ML Inc.（Infinia ML公司）；Duke University（杜克大学）
We propose a scalable Gromov-Wasserstein learning (S-GWL) method and establish a novel and theoretically-supported paradigm for large-scale graph analysis. The proposed method is based on the fact that Gromov-Wasserstein discrepancy is a pseudometric on graphs. Given two graphs, the optimal transport associated with their Gromov-Wasserstein discrepancy provides the correspondence between their nodes and achieves graph matching. When one of the graphs has isolated but self-connected nodes ($i.e.$, a disconnected graph), the optimal transport indicates the clustering structure of the other graph…
Télécom Paris（巴黎电信学院）；Institut Polytechnique de Paris（巴黎理工学院）；Ecole Polytechnique（巴黎综合理工学院）；Université de Toulouse（图卢兹大学）；INRAE（法国国家农业食品与环境研究院）；Aalto University（阿尔托大学）
This paper introduces a novel and generic framework to solve the flagship task of supervised labeled graph prediction by leveraging Optimal Transport tools. We formulate the problem as regression with the Fused Gromov-Wasserstein (FGW) loss and propose a predictive model relying on a FGW barycenter whose weights depend on inputs. First we introduce a non-parametric estimator based on kernel ridge regression for which theoretical results such as consistency and excess risk bound are proved. Next we propose an interpretable parametric model where the barycenter weights are modeled with a neural …
In the context of optimal transport methods, the subspace detour approach was recently presented by Muzellec and Cuturi (2019). It consists in building a nearly optimal transport plan in the measures space from an optimal transport plan in a wisely chosen subspace, onto which the original measures are projected. The contribution of this paper is to extend this category of methods to the Gromov-Wasserstein problem, which is a particular type of transport distance involving the inner geometry of the compared distributions. After deriving the associated formalism and properties, we also discuss a…
Distributional Reduction: Unifying Dimensionality Reduction and Clustering with Gromov-Wasserstein
ENS de Lyon, CNRS, UMPA UMR 5669（里昂大学、国家科学研究中心、UMPA UMR 5669）；EPFL, LTS4（苏黎世联邦理工学院、LTS4）；Université Bretagne Sud, IRISA UMR 6074（布列塔尼南部大学、IRISA UMR 6074）；École polytechnique, Institut Polytechnique de Paris, CMAP UMR 7641（巴黎高等学院、巴黎理工学院、CMAP UMR 7641）；Inria, ENS de Lyon, CNRS, Université Claude Bernard Lyon 1, LIP UMR 5668（法国国家科学研究中心、里昂大学、 Claude Bernard Lyon 1 大学、LIP UMR 5668）
Gromov-Wasserstein-like Distances in the Gaussian Mixture Models Space
ENS Paris-Saclay（巴黎萨克雷高等师范学校）；CNRS（法国国家科学研究中心）；Centre Borelli（博雷利中心）；Université de Paris（巴黎大学）；MAP5（数学及其应用实验室）；Institut Universitaire de France（法国大学研究院）
Structure-Preserving Multi-View Embedding Using Gromov-Wasserstein Optimal Transport
Instituto Federal de Educação, Ciência e Tecnologia do Ceará（塞阿拉联邦教育、科学与技术学院）；Sigma Nova；Department of Teleinformatics Engineering, Federal University of Ceará（塞阿拉联邦大学远程信息工程系）
The ability to align points across two related yet incomparable point clouds (e.g. living in different spaces) plays an important role in machine learning. The Gromov-Wasserstein (GW) framework provides an increasingly popular answer to such problems, by seeking a low-distortion, geometry-preserving assignment between these points. As a non-convex, quadratic generalization of optimal transport (OT), GW is NP-hard. While practitioners often resort to solving GW approximately as a nested sequence of entropy-regularized OT problems, the cubic complexity (in the number $n$ of samples) of that appr…
Univ. Côte d’Azur（蔚蓝海岸大学）；Inria（法国国家信息与自动化研究所）；CNRS（法国国家科学研究中心）；IP Paris（巴黎理工学院）；CMAP（应用数学中心）；MSI（数学科学研究所）；Univ. Lyon（里昂大学）；ENS de Lyon（里昂高等师范学院）；IRISA（法国信息与随机系统研究所）
Comparing structured objects such as graphs is a fundamental operation involved in many learning tasks. To this end, the Gromov-Wasserstein (GW) distance, based on Optimal Transport (OT), has proven to be successful in handling the specific nature of the associated objects. More specifically, through the nodes connectivity relations, GW operates on graphs, seen as probability measures over specific spaces. At the core of OT is the idea of conservation of mass, which imposes a coupling between all the nodes from the two considered graphs. We argue in this paper that this property can be detrime…
Stanford University（斯坦福大学）；University of Utah（犹他大学）；Florida State University（佛罗里达州立大学）
The Gromov-Wasserstein (GW) framework adapts ideas from optimal transport to allow for the comparison of probability distributions defined on different metric spaces. Scalable computation of GW distances and associated matchings on graphs and point clouds have recently been made possible by state-of-the-art algorithms such as S-GWL and MREC. Each of these algorithmic breakthroughs relies on decomposing the underlying spaces into parts and performing matchings on these parts, adding recursion as needed. While very successful in practice, theoretical guarantees on such methods are limited. Inspi…
Stanford University（斯坦福大学）；Florida State University（佛罗里达州立大学）
We establish a bridge between spectral clustering and Gromov-Wasserstein Learning (GWL), a recent optimal transport-based approach to graph partitioning. This connection both explains and improves upon the state-of-the-art performance of GWL. The Gromov-Wasserstein framework provides probabilistic correspondences between nodes of source and target graphs via a quadratic programming relaxation of the node matching problem. Our results utilize and connect the observations that the GW geometric structure remains valid for any rank-2 tensor, in particular the adjacency, distance, and various kerne…
Classical optimal transport problem seeks a transportation map that preserves the total mass betwenn two probability distributions, requiring their mass to be the same. This may be too restrictive in certain applications such as color or shape matching, since the distributions may have arbitrary masses and/or that only a fraction of the total mass has to be transported. Several algorithms have been devised for computing partial Wasserstein metrics that rely on an entropic regularization, but when it comes with exact solutions, almost no partial formulation of neither Wasserstein nor Gromov-Was…
Fused Partial Gromov-Wasserstein for Structured Objects
Department of Computer Science, Vanderbilt University（范德比大学计算机科学系）；Department of Computer Science, Indiana University–Indianapolis（印第安纳大学印第安纳市计算机科学系）；Department of BioHealth Informatics, Indiana University–Indianapolis（印第安纳大学印第安纳市生物健康信息学系）；Department of Mathematics and Computer Science, Fisk University（菲斯克大学数学与计算机科学系）
Covariance alignment: from maximum likelihood estimation to Gromov-Wasserstein
Center for Data Science, New York University（纽约大学数据科学中心）；Institute for Data, Systems, and Society, Massachusetts Institute of Technology（麻省理工学院数据、系统与社会研究所）；Department of Mathematics, Massachusetts Institute of Technology（麻省理工学院数学系）
Fused Gromov-Wasserstein Distance with Feature Selection
School of Data Science and Society（数据科学与社会学院）；Division of Computational Social Science（计算社会科学系）；University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Courant Institute（Courant研究所）；Department of Mathematics（数学系）；New York University（纽约大学）；Duke University（杜克大学）
Fused Gromov-Wasserstein Graph Mixup for Graph-level Classifications
Peking University（北京大学）；Tsinghua University（清华大学）；National Research and Engineering Center of Software Engineering, Peking University（北京大学软件工程国家工程研究中心）
THESAURUS: Contrastive Graph Clustering by Swapping Fused Gromov-Wasserstein Couplings
School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机科学与工程学院）；School of Systems Science and Engineering, Sun Yat-sen University（中山大学系统科学与工程学院）
Optimal transport tools (OTT-JAX) is a Python toolbox that can solve optimal transport problems between point clouds and histograms. The toolbox builds on various JAX features, such as automatic and custom reverse mode differentiation, vectorization, just-in-time compilation and accelerators support. The toolbox covers elementary computations, such as the resolution of the regularized OT problem, and more advanced extensions, such as barycenters, Gromov-Wasserstein, low-rank solvers, estimation of convex maps, differentiable generalizations of quantiles and ranks, and approximate OT between Ga…
This work considers the problem of computing distances between structured objects such as undirected graphs, seen as probability distributions in a specific metric space. We consider a new transportation distance (i.e. that minimizes a total cost of transporting probability masses) that unveils the geometric nature of the structured objects space. Unlike Wasserstein or Gromov-Wasserstein metrics that focus solely and respectively on features (by considering a metric in the feature space) or structure (by seeing structure as a metric space), our new distance exploits jointly both information, a…
Dynamic Generalized Gromov-Wasserstein Optimal Transport
Beijing International Center for Mathematical Research, Peking University（北京大学北京国际数学研究中心）；Center for Quantitative Biology, Peking University（北京大学定量生物学中心）；Center for Machine Learning Research, Peking University（北京大学机器学习研究中心）；National Engineering Laboratory for Big Data Analysis and Applications, Beijing（北京大数据分析与应用国家工程实验室）；AI for Science Institute, Beijing（北京科学智能研究院）；School of Mathematical Sciences, Peking University（北京大学数学科学学院）；Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）
Gromov-Wasserstein and optimal transport: from assignment problems to probabilistic numeric
University of Milano-Bicocca, Department of Computer Science Systems and Communication（米兰-比科卡大学计算机科学与通信系）；University of Milano-Bicocca, Department of Economics Management and Statistics（米兰-比科卡大学经济学管理与统计系）
Metric properties of partial and robust Gromov-Wasserstein distances
University of Houston（休斯顿大学）；Colorado School of Mines（科罗拉多矿业学院）；University of Maryland（马里兰大学）；University of Nebraska - Lincoln（内布拉斯加大学林肯分校）；University of Tennessee - Knoxville（田纳西大学诺克斯维尔分校）；Florida State University（佛罗里达州立大学）；Johns Hopkins University（约翰斯·霍普金斯大学）
IBM Research（IBM研究院）；IBM T.J Watson Research Center（IBM托马斯·J·沃森研究中心）
We propose Gaussian optimal transport for Image style transfer in an Encoder/Decoder framework. Optimal transport for Gaussian measures has closed forms Monge mappings from source to target distributions. Moreover interpolates between a content and a style image can be seen as geodesics in the Wasserstein Geometry. Using this insight, we show how to mix different target styles , using Wasserstein barycenter of Gaussian measures. Since Gaussians are closed under Wasserstein barycenter, this allows us a simple style transfer and style mixing and interpolation. Moreover we show how mixing differe…
Comparing structured data from possibly different metric-measure spaces is a fundamental task in machine learning, with applications in, e.g., graph classification. The Gromov-Wasserstein (GW) discrepancy formulates a coupling between the structured data based on optimal transportation, tackling the incomparability between different structures by aligning the intra-relational geometries. Although efficient \emph{local} solvers such as conditional gradient and Sinkhorn are available, the inherent non-convexity still prevents a tractable evaluation, and the existing lower bounds are not tight en…
VinAI Research（VinAI研究院）；University of Texas, Austin（德克萨斯大学奥斯汀分校）
Relational regularized autoencoder (RAE) is a framework to learn the distribution of data by minimizing a reconstruction loss together with a relational regularization on the latent space. A recent attempt to reduce the inner discrepancy between the prior and aggregated posterior distributions is to incorporate sliced fused Gromov-Wasserstein (SFG) between these distributions. That approach has a weakness since it treats every slicing direction similarly, meanwhile several directions are not useful for the discriminative task. To improve the discrepancy and consequently the relational regulari…
We propose a new nonlinear factorization model for graphs that are with topological structures, and optionally, node attributes. This model is based on a pseudometric called Gromov-Wasserstein (GW) discrepancy, which compares graphs in a relational way. It estimates observed graphs as GW barycenters constructed by a set of atoms with different weights. By minimizing the GW discrepancy between each observed graph and its GW barycenter-based estimation, we learn the atoms and their weights associated with the observed graphs. The model achieves a novel and flexible factorization mechanism under …
We propose a novel and principled method to learn a nonparametric graph model called graphon, which is defined in an infinite-dimensional space and represents arbitrary-size graphs. Based on the weak regularity lemma from the theory of graphons, we leverage a step function to approximate a graphon. We show that the cut distance of graphons can be relaxed to the Gromov-Wasserstein distance of their step functions. Accordingly, given a set of graphs generated by an underlying graphon, we learn the corresponding step function as the Gromov-Wasserstein barycenter of the given graphs. Furthermore, …
University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；Google（谷歌）
Based on the concepts of Wasserstein barycenter (WB) and Gromov-Wasserstein barycenter (GWB), we propose a unified mathematical framework for neural network (NN) model fusion and utilize it to reveal new insights about the linear mode connectivity of SGD solutions. In our framework, the fusion occurs in a layer-wise manner and builds on an interpretation of a node in a network as a function of the layer preceding it. The versatility of our mathematical framework allows us to talk about model fusion and linear mode connectivity for a broad class of NNs, including fully connected NN, CNN, ResNet…
