最优传输（OT）因其在机器学习、统计学和信号处理等领域的广泛应用而备受关注。然而，平衡质量的要求限制了它在实际问题中的性能。为了解决这些限制，人们提出了 OT 问题的多种变体，包括不平衡 OT、最优部分传输（OPT）和 Hellinger Kantorovich（HK）。在本文中，我们提出了线性最优部分传输（LOPT）嵌入，它将 OT 和 HK 上的（局部）线性化技术推广到了 OPT 问题。所提出的嵌入方法能够更快地计算成对正测度之间的 OPT 距离。除了理论贡献外，我们还在点云插值和 PCA 分析中展示了 LOPT 嵌入技术。
英文摘要：
Optimal transport (OT) has gained popularity due to its various applications in fields such as machine learning, statistics, and signal processing. However, the balanced mass requirement limits its performance in practical problems. To address these limitations, variants of the OT problem, including unbalanced OT, Optimal partial transport (OPT), and Hellinger Kantorovich (HK), have been proposed. In this paper, we propose the Linear optimal partial transport (LOPT) embedding, which extends the (local) linearization technique on OT and HK to the OPT problem. The proposed embedding allows for faster computation of OPT distance between pairs of positive measures. Besides our theoretical contributions, we demonstrate the LOPT embedding technique in point-cloud interpolation and PCA analysis.
Indian Institute of Technology Hyderabad（印度理工学院海得拉巴分校）；Microsoft IDC（微软IDC）
The objective in statistical Optimal Transport (OT) is to consistently estimate the optimal transport plan/map solely using samples from the given source and target marginal distributions. This work takes the novel approach of posing statistical OT as that of learning the transport plan's kernel mean embedding from sample based estimates of marginal embeddings. The proposed estimator controls overfitting by employing maximum mean discrepancy based regularization, which is complementary to $ϕ$-divergence (entropy) based regularization popularly employed in existing estimators. A key result is t…
Embedding Empirical Distributions for Computing Optimal Transport Maps
Hong Kong Baptist University（香港 Baptist 大学）；Institute of Science Tokyo（东京科学研究所）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of Southern California（南加州大学）；University of Maryland, College Park（马里兰大学学院公园分校）
University of California, San Diego（加州大学圣地亚哥分校）；Halicioğlu Data Science Institute（哈利西奥卢数据科学研究所）
Discriminating between distributions is an important problem in a number of scientific fields. This motivated the introduction of Linear Optimal Transportation (LOT), which embeds the space of distributions into an $L^2$-space. The transform is defined by computing the optimal transport of each distribution to a fixed reference distribution, and has a number of benefits when it comes to speed of computation and to determining classification boundaries. In this paper, we characterize a number of settings in which LOT embeds families of distributions into a space in which they are linearly separ…
Constituting highly informative network embeddings is an important tool for network analysis. It encodes network topology, along with other useful side information, into low-dimensional node-based feature representations that can be exploited by statistical modeling. This work focuses on learning context-aware network embeddings augmented with text data. We reformulate the network-embedding problem, and present two novel strategies to improve over traditional attention mechanisms: ($i$) a content-aware sparse attention module based on optimal transport, and ($ii$) a high-level attention parsin…
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；VinAI Research（VinAI研究院）
Mini-batch optimal transport (m-OT) has been widely used recently to deal with the memory issue of OT in large-scale applications. Despite their practicality, m-OT suffers from misspecified mappings, namely, mappings that are optimal on the mini-batch level but are partially wrong in the comparison with the optimal transportation plan between the original measures. Motivated by the misspecified mappings issue, we propose a novel mini-batch method by using partial optimal transport (POT) between mini-batch empirical measures, which we refer to as mini-batch partial optimal transport (m-POT). Le…
Structure-Preserving Multi-View Embedding Using Gromov-Wasserstein Optimal Transport
Instituto Federal de Educação, Ciência e Tecnologia do Ceará（塞阿拉联邦教育、科学与技术学院）；Sigma Nova；Department of Teleinformatics Engineering, Federal University of Ceará（塞阿拉联邦大学远程信息工程系）
Université de Technologie de Compiègne（贡比涅技术大学）；Université de Rouen Normandie（鲁昂诺曼底大学）；INSA Rouen（鲁昂国立应用科学学院）；Criteo AI Lab（Criteo AI实验室）
We propose a novel approach for comparing distributions whose supports do not necessarily lie on the same metric space. Unlike Gromov-Wasserstein (GW) distance which compares pairwise distances of elements from each distribution, we consider a method allowing to embed the metric measure spaces in a common Euclidean space and compute an optimal transport (OT) on the embedded distributions. This leads to what we call a sub-embedding robust Wasserstein (SERW) distance. Under some conditions, SERW is a distance that considers an OT distance of the (low-distorted) embedded distributions using a com…
University of Edinburgh（爱丁堡大学）；École Normale Supérieure（巴黎高等师范学院）；CNRS（法国国家科学研究中心）
Optimal transport distances (OT) have been widely used in recent work in Machine Learning as ways to compare probability distributions. These are costly to compute when the data lives in high dimension. Recent work by Paty et al., 2019, aims specifically at reducing this cost by computing OT using low-rank projections of the data (seen as discrete measures). We extend this approach and show that one can approximate OT distances by using more general families of maps provided they are 1-Lipschitz. The best estimate is obtained by maximising OT over the given family. As OT calculations are done …
Neighbor Embeddings Using Unbalanced Optimal Transport Metrics
Department of Mathematics, University of Texas at Arlington（德克萨斯大学阿灵顿分校数学系）；Division of Data Science, University of Texas at Arlington（德克萨斯大学阿灵顿分校数据科学 division）
We address the problem of learning on sets of features, motivated by the need of performing pooling operations in long biological sequences of varying sizes, with long-range dependencies, and possibly few labeled data. To address this challenging task, we introduce a parametrized representation of fixed size, which embeds and then aggregates elements from a given input set according to the optimal transport plan between the set and a trainable reference. Our approach scales to large datasets and allows end-to-end training of the reference, while also providing a simple unsupervised learning me…
Classical optimal transport problem seeks a transportation map that preserves the total mass betwenn two probability distributions, requiring their mass to be the same. This may be too restrictive in certain applications such as color or shape matching, since the distributions may have arbitrary masses and/or that only a fraction of the total mass has to be transported. Several algorithms have been devised for computing partial Wasserstein metrics that rely on an entropic regularization, but when it comes with exact solutions, almost no partial formulation of neither Wasserstein nor Gromov-Was…
Laboratoire de Mathématiques d’Orsay（奥赛数学实验室）；Univ. Paris-Sud（巴黎第十一大学）；CNRS（法国国家科学研究中心）；Université Paris-Saclay（巴黎萨克雷大学）；Inria Saclay - Ile-de-France（法国国家信息与自动化研究所萨克雷-法兰西岛分部）
This work studies an explicit embedding of the set of probability measures into a Hilbert space, defined using optimal transport maps from a reference probability density. This embedding linearizes to some extent the 2-Wasserstein space, and enables the direct use of generic supervised and unsupervised learning algorithms on measure data. Our main result is that the embedding is (bi-)Hölder continuous, when the reference density is uniform over a convex set, and can be equivalently phrased as a dimension-independent Hölder-stability results for optimal transport maps.
University of Texas, Austin（德克萨斯大学奥斯汀分校）；VinAI Research（VinAI研究院）
We study the multi-marginal partial optimal transport (POT) problem between $m$ discrete (unbalanced) measures with at most $n$ supports. We first prove that we can obtain two equivalence forms of the multimarginal POT problem in terms of the multimarginal optimal transport problem via novel extensions of cost tensor. The first equivalence form is derived under the assumptions that the total masses of each measure are sufficiently close while the second equivalence form does not require any conditions on these masses but at the price of more sophisticated extended cost tensor. Our proof techni…
Entropic Optimal Transport Eigenmaps for Nonlinear Alignment and Joint Embedding of High-Dimensional Datasets
Department of Electrical and Computer Engineering, Yale University（耶鲁大学电气与计算机工程系）；Department of Biostatistics, Harvard University（哈佛大学生物统计学系）；Program in Applied Mathematics, Yale University（耶鲁大学应用数学项目）；Interdepartmental Program in Computational Biology and Bioinformatics, Yale University（耶鲁大学计算生物学与生物信息学跨学科项目）；Department of Pathology, Yale University School of Medicine（耶鲁大学医学院病理学系）
The current best practice for computing optimal transport (OT) is via entropy regularization and Sinkhorn iterations. This algorithm runs in quadratic time as it requires the full pairwise cost matrix, which is prohibitively expensive for large sets of objects. In this work we propose two effective log-linear time approximations of the cost matrix: First, a sparse approximation based on locality-sensitive hashing (LSH) and, second, a Nyström approximation with LSH-based sparse corrections, which we call locally corrected Nyström (LCN). These approximations enable general log-linear time algori…
Wasserstein Wormhole: Scalable Optimal Transport Distance with Transformers
Sloan Kettering Institute（斯隆凯特琳研究所）；Memorial Sloan Kettering Cancer Center（纪念斯隆凯特琳癌症中心）；Tri-Institutional Training Program in Computational Biology and Medicine（三机构计算生物学与医学培训项目）；Weill Cornell Medicine（威尔康奈尔医学院）；Columbia University（哥伦比亚大学）；Yale University（耶鲁大学）；Howard Hughes Medical Institute（霍华德·休斯医学研究所）
Infinia ML, Inc.（Infinia ML公司）；Duke University（杜克大学）；Tencent AI Lab（腾讯人工智能实验室）
We propose a novel Wasserstein method with a distillation mechanism, yielding joint learning of word embeddings and topics. The proposed method is based on the fact that the Euclidean distance between word embeddings may be employed as the underlying distance in the Wasserstein topic model. The word distributions of topics, their optimal transports to the word distributions of documents, and the embeddings of words are learned in a unified framework. When learning the topic model, we leverage a distilled underlying distance matrix to update the topic distributions and smoothly calculate the co…
Stochastic neighbor embedding (SNE) and related nonlinear manifold learning algorithms achieve high-quality low-dimensional representations of similarity data, but are notoriously slow to train. We propose a generic formulation of embedding algorithms that includes SNE and other existing algorithms, and study their relation with spectral methods and graph Laplacians. This allows us to define several partial-Hessian optimization strategies, characterize their global and local convergence, and evaluate them empirically. We achieve up to two orders of magnitude speedup over existing training meth…
Faculty of Computer Science and Management（计算机科学与管理学院）
The role of spatial data in tackling city-related tasks has been growing in recent years. To use them in machine learning models, it is often necessary to transform them into a vector representation, which has led to the development in the field of spatial data representation learning. There is also a growing variety of spatial data types for which representation learning methods are proposed. Public transport timetables have so far not been used in the task of learning representations of regions in a city. In this work, a method is developed to embed public transport availability information …
Facebook AI（脸书人工智能）；Santa Clara University（圣克拉拉大学）；Google Research（谷歌研究院）；Meituan NLP Center（美团自然语言处理中心）；Sun Yat-sen University（中山大学）；Rochester Institute of Technology（罗切斯特理工学院）
Network embedding is an effective technique to learn the low-dimensional representations of nodes in networks. Real-world networks are usually with multiplex or having multi-view representations from different relations. Recently, there has been increasing interest in network embedding on multiplex data. However, most existing multiplex approaches assume that the data is complete in all views. But in real applications, it is often the case that each view suffers from the missing of some data and therefore results in partial multiplex data. In this paper, we present a novel Deep Partial Multipl…
Wrocław University of Science and Technology（弗罗茨瓦夫理工大学）
We selected 48 European cities and gathered their public transport timetables in the GTFS format. We utilized Uber's H3 spatial index to divide each city into hexagonal micro-regions. Based on the timetables data we created certain features describing the quantity and variety of public transport availability in each region. Next, we trained an auto-associative deep neural network to embed each of the regions. Having such prepared representations, we then used a hierarchical clustering approach to identify similar regions. To do so, we utilized an agglomerative clustering algorithm with a eucli…
Deep embedding based text-independent speaker verification has demonstrated superior performance to traditional methods in many challenging scenarios. Its loss functions can be generally categorized into two classes, i.e., verification and identification. The verification loss functions match the pipeline of speaker verification, but their implementations are difficult. Thus, most state-of-the-art deep embedding methods use the identification loss functions with softmax output units or their variants. In this paper, we propose a verification loss function, named the maximization of partial are…
CSAIL, Massachusetts Institute of Technology（麻省理工学院计算机科学与人工智能实验室）
Current graph neural network (GNN) architectures naively average or sum node embeddings into an aggregated graph representation -- potentially losing structural or semantic information. We here introduce OT-GNN, a model that computes graph embeddings using parametric prototypes that highlight key facets of different graph aspects. Towards this goal, we successfully combine optimal transport (OT) with parametric graph models. Graph representations are obtained from Wasserstein distances between the set of GNN node embeddings and ``prototype'' point clouds as free parameters. We theoretically pr…
University of Rochester（罗切斯特大学）；KAUST（阿卜杜拉国王科技大学）；Iowa State University（爱荷华州立大学）；University of Michigan（密歇根大学）；PlusAI（普拉斯人工智能公司）；Kwai Inc.（快手公司）
Graph node embedding aims at learning a vector representation for all nodes given a graph. It is a central problem in many machine learning tasks (e.g., node classification, recommendation, community detection). The key problem in graph node embedding lies in how to define the dependence to neighbors. Existing approaches specify (either explicitly or implicitly) certain dependencies on neighbors, which may lead to loss of subtle but important structural information within the graph and other dependencies among neighbors. This intrigues us to ask the question: can we design a model to give the …
College of Software Engineering, Southeast University（东南大学软件学院）；School of Computer Science and Engineering, Southeast University（东南大学计算机科学与工程学院）；Yam Pak Charitable Foundation School of Computing and Information Sciences, Saint Francis University（圣弗朗西斯大学叶荫堂计算与信息科学学院）；Department of Computer Science, City University of Hong Kong（香港城市大学计算机科学系）
