We compare several approaches to learn an Optimal Map, represented as a neural network, between probability distributions. The approaches fall into two categories: ``Heuristics'' and approaches with a more sound mathematical justification, motivated by the dual of the Kantorovitch problem. Among the algorithms we consider a novel approach involving dynamic flows and reductions of Optimal Transport to supervised learning.
Skolkovo Institute of Science and Technology（斯科尔科沃科学技术学院）；Artificial Intelligence Research Institute（人工智能研究所）
We study the Neural Optimal Transport (NOT) algorithm which uses the general optimal transport formulation and learns stochastic transport plans. We show that NOT with the weak quadratic cost might learn fake plans which are not optimal. To resolve this issue, we introduce kernel weak quadratic costs. We show that they provide improved theoretical guarantees and practical performance. We test NOT with kernel costs on the unpaired image-to-image translation task.
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of California, Irvine（加州大学欧文分校）；Princeton University（普林斯顿大学）；University of Washington（华盛顿大学）
In this paper, we present a novel and principled approach to learn the optimal transport between two distributions, from samples. Guided by the optimal transport theory, we learn the optimal Kantorovich potential which induces the optimal transport map. This involves learning two convex functions, by solving a novel minimax optimization. Building upon recent advances in the field of input convex neural networks, we propose a new framework where the gradient of one convex function represents the optimal transport mapping. Numerical experiments confirm that we learn the optimal transport mapping…
Skolkovo Institute of Science and Technology（斯科尔科沃科学技术学院）；Artificial Intelligence Research Institute（人工智能研究所）
We present a novel neural-networks-based algorithm to compute optimal transport maps and plans for strong and weak transport costs. To justify the usage of neural networks, we prove that they are universal approximators of transport plans between probability distributions. We evaluate the performance of our optimal transport algorithm on toy examples and on the unpaired image-to-image translation.
Neural Optimal Transport Meets Multivariate Conformal Prediction
Department of ML, MBZUAI（机器学习系，MBZUAI）；MBZUAI, UAE（MBZUAI，阿联酋）；CMAP, École polytechnique（CMAP，巴黎高等理工学院）；Ecole Polytechnique, Palaiseau, France（巴黎高等理工学院，Palaiseau，法国）
The University of Tokyo（东京大学）；RIKEN Center for Advanced Intelligence Project（理化学研究所先进智能项目中心）
We introduce a new theoretical framework to analyze deep learning optimization with connection to its generalization error. Existing frameworks such as mean field theory and neural tangent kernel theory for neural network optimization analysis typically require taking limit of infinite width of the network to show its global convergence. This potentially makes it difficult to directly deal with finite width network; especially in the neural tangent kernel regime, we cannot reveal favorable properties of neural networks beyond kernel methods. To realize more natural analysis, we consider a comp…
Imperial College London, London, UK（伦敦帝国理工学院）；University Of Copenhagen, Copenhagen, Denmark（哥本哈根大学）；KTH Royal Institute of Technology, Stockholm, Sweden（瑞典皇家理工学院）
Skolkovo Institute of Science and Technology（斯科尔科沃科学技术学院）；Massachusetts Institute of Technology（麻省理工学院）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）；Artificial Intelligence Research Institute（人工智能研究所）
Despite the recent popularity of neural network-based solvers for optimal transport (OT), there is no standard quantitative way to evaluate their performance. In this paper, we address this issue for quadratic-cost transport -- specifically, computation of the Wasserstein-2 distance, a commonly-used formulation of optimal transport in machine learning. To overcome the challenge of computing ground truth transport maps between continuous measures needed to assess these solvers, we use input-convex neural networks (ICNN) to construct pairs of measures whose ground truth OT maps can be obtained a…
Computing high-dimensional optimal transport by flow neural networks
H. Milton Stewart School of Industrial and Systems Engineering, Georgia Institute of Technology（佐治亚理工学院H. Milton Stewart工业与系统工程学院）；Department of Mathematics, Duke University（杜克大学数学系）
Université de Bretagne Sud（南布列塔尼大学）；IRISA（法国信息与随机系统研究所）；CNRS（法国国家科学研究中心）；Université Côte d’Azur（蔚蓝海岸大学）；Kyoto University（京都大学）
Deep neural networks have established as a powerful tool for large scale supervised classification tasks. The state-of-the-art performances of deep neural networks are conditioned to the availability of large number of accurately labeled samples. In practice, collecting large scale accurately labeled datasets is a challenging and tedious task in most scenarios of remote sensing image analysis, thus cheap surrogate procedures are employed to label the dataset. Training deep neural networks on such datasets with inaccurate labels easily overfits to the noisy training labels and degrades the perf…
University of Utah（犹他大学）；Florida State University（佛罗里达州立大学）
Optimal Mass Transport (OMT) is a well studied problem with a variety of applications in a diverse set of fields ranging from Physics to Computer Vision and in particular Statistics and Data Science. Since the original formulation of Monge in 1781 significant theoretical progress been made on the existence, uniqueness and properties of the optimal transport maps. The actual numerical computation of the transport maps, particularly in high dimensions, remains a challenging problem. By Brenier's theorem, the continuous OMT problem can be reduced to that of solving a non-linear PDE of Monge-Amper…
Harvard John A. Paulson School of Engineering and Applied Sciences（哈佛大学约翰·A·保罗森工程与应用科学学院）；Kempner Institute at Harvard University（哈佛大学凯门研究所）；Weierstrass Institute, Berlin, Germany（魏尔斯特拉斯研究所）
Neural Optimal Transport with General Cost Functionals
Artificial Intelligence Research Institute（人工智能研究所）；Skolkovo Institute of Science and Technology（斯科尔科沃科学技术研究所）；Moscow Institute of Physics and Technology（莫斯科物理技术学院）；HSE University（高等经济大学）；Yandex（Yandex（俄罗斯互联网公司））
Department of Management, Economics and Industrial Engineering, Politecnico di Milano（米兰理工大学管理、经济与工业工程系）；RFF-CMCC European Institute on Economics and the Environment (EIEE)（RFF-CMCC欧洲经济与环境研究所）；Euro-Mediterranean Center on Climate Change (CMCC)（欧洲-地中海气候变化中心）；Bocconi University, Department of Decision Sciences（博科尼大学决策科学系）；MIT, Department of Nuclear Science and Engineering（麻省理工学院核科学与工程系）
ETH Zurich（苏黎世联邦理工学院）；University of Zurich（苏黎世大学）；Microsoft Research（微软研究院）
Comparing unpaired samples of a distribution or population taken at different points in time is a fundamental task in many application domains where measuring populations is destructive and cannot be done repeatedly on the same sample, such as in single-cell biology. Optimal transport (OT) can solve this challenge by learning an optimal coupling of samples across distributions from unpaired data. However, the usual formulation of OT assumes conservation of mass, which is violated in unbalanced scenarios in which the population size changes (e.g., cell proliferation or death) between measuremen…
CloneOT: predicting clone-specific perturbation responses with lineage-supervised neural inverse optimal transport
University of Maryland（马里兰大学）；Center of Bioinformatics and Computational Biology（生物信息学与计算生物学中心）；University of Pittsburgh（匹兹堡大学）；UPMC Children’s Hospital of Pittsburgh（匹兹堡UPMC儿童医院）；Institute of Health Computing（健康计算研究所）
Institut de Mathématiques de Toulouse（图卢兹数学研究所）；CNRS（法国国家科学研究中心）；Université de Toulouse（图卢兹大学）；Institut de Recherche en Informatique de Toulouse（图卢兹计算机科学研究所）；AKKODIS group（AKKODIS集团）
Automatic recommendation systems based on deep neural networks have become extremely popular during the last decade. Some of these systems can however be used for applications which are ranked as High Risk by the European Commission in the A.I. act, as for instance for online job candidate recommendation. When used in the European Union, commercial AI systems for this purpose will then be required to have to proper statistical properties with regard to potential discrimination they could engender. This motivated our contribution, where we present a novel optimal transport strategy to mitigate …
Recently, Neural Topic Models (NTMs) inspired by variational autoencoders have obtained increasingly research interest due to their promising results on text analysis. However, it is usually hard for existing NTMs to achieve good document representation and coherent/diverse topics at the same time. Moreover, they often degrade their performance severely on short documents. The requirement of reparameterisation could also comprise their training quality and model flexibility. To address these shortcomings, we present a new neural topic model via the theory of optimal transport (OT). Specificall…
Moscow Independent Research Institute of Artificial Intelligence（莫斯科独立人工智能研究院）；Applied AI Institute（应用人工智能研究所）；Lomonosov Moscow State University（莫斯科国立罗蒙诺索夫大学）；MIRIAI
Neural Optimal Transport in Hilbert Spaces: Characterizing Spurious Solutions and Gaussian Smoothing
Department of Statistics, Sungkyunkwan University, Seoul, Republic of Korea（统计系，顺天大学，首尔，大韩民国）；Department of Mathematics, University of Seoul, Seoul, Republic of Korea（数学系，首尔大学，首尔，大韩民国）；Korea Institute for Advanced Study, Seoul, Republic of Korea（韩国高级研究院，首尔，大韩民国）
Department of Mathematics, Columbia University（哥伦比亚大学数学系）；HKU Musketeers Foundation Institute of Data Science（香港大学 Musketeers 基金会数据科学研究所）；HKU-Shanghai Intelligent Computing Research Center（香港大学-上海智能计算研究中心）
We introduce a novel and highly tractable supervised learning approach based on neural networks that can be applied for the computation of model-free price bounds of, potentially high-dimensional, financial derivatives and for the determination of optimal hedging strategies attaining these bounds. In particular, our methodology allows to train a single neural network offline and then to use it online for the fast determination of model-free price bounds of a whole class of financial derivatives with current market data. We show the applicability of this approach and highlight its accuracy in s…
Monge map refers to the optimal transport map between two probability distributions and provides a principled approach to transform one distribution to another. Neural network based optimal transport map solver has gained great attention in recent years. Along this line, we present a scalable algorithm for computing the neural Monge map between two probability distributions. Our algorithm is based on a weak form of the optimal transport problem, thus it only requires samples from the marginals instead of their analytic expressions, and can accommodate optimal transport between two distribution…
Implicit Neural Optimal Transport via Fixed-Point Optimization
Department of Mathematics, University of California, Los Angeles（加州大学洛杉矶分校数学系）；Department of Applied Mathematics and Statistics, Colorado School of Mines（科罗拉多矿业学院应用数学与统计系）
On the explainable properties of 1-Lipschitz Neural Networks: An Optimal Transport Perspective
Université Paul-Sabatier（保罗萨巴蒂尔大学）；Institut de Recherche Technologique Saint-Exupéry（圣埃克苏佩里技术研究院）；Carney Institute for Brain Science,Brown University（布朗大学卡尼脑科学研究所）；Innovation & Research Division, SNCF（法国国家铁路公司创新与研究部）
Optimal Transport for Unsupervised Hallucination Detection in Neural Machine Translation
Instituto de Telecomunicações（电信研究所）；Instituto Superior Técnico（高等技术学院）；University of Lisbon（里斯本大学）；CentraleSupélec（中央理工-高等电力学院）；Université Paris-Saclay（巴黎萨克雷大学）；CNRS（法国国家科学研究中心）；Unbabel（Unbabel公司）
National Institute of Information and Communications Technology（日本国立信息与通信技术研究所）；Research Center for Information Technology Innovation, Academic Sinica（中央研究院资讯科技创新研究中心）
Due to the mismatch of statistical distributions of acoustic speech between training and testing sets, the performance of spoken language identification (SLID) could be drastically degraded. In this paper, we propose an unsupervised neural adaptation model to deal with the distribution mismatch problem for SLID. In our model, we explicitly formulate the adaptation as to reduce the distribution discrepancy on both feature and classifier for training and testing data sets. Moreover, inspired by the strong power of the optimal transport (OT) to measure distribution discrepancy, a Wasserstein dist…
The feature map obtained from the denoising autoencoder (DAE) is investigated by determining transportation dynamics of the DAE, which is a cornerstone for deep learning. Despite the rapid development in its application, deep neural networks remain analytically unexplained, because the feature maps are nested and parameters are not faithful. In this paper, we address the problem of the formulation of nested complex of parameters by regarding the feature map as a transport map. Even when a feature map has different dimensions between input and output, we can regard it as a transportation map by…
Neural networks trained with backpropagation, the standard algorithm of deep learning which uses weight transport, are easily fooled by existing gradient-based adversarial attacks. This class of attacks are based on certain small perturbations of the inputs to make networks misclassify them. We show that less biologically implausible deep neural networks trained with feedback alignment, which do not use weight transport, can be harder to fool, providing actual robustness. Tested on MNIST, deep neural networks trained without weight transport (1) have an adversarial accuracy of 98% compared to …
University of Illinois, Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
This paper establishes rates of universal approximation for the shallow neural tangent kernel (NTK): network weights are only allowed microscopic changes from random initialization, which entails that activations are mostly unchanged, and the network is nearly equivalent to its linearization. Concretely, the paper has two main contributions: a generic scheme to approximate functions with the NTK by sampling from transport mappings between the initial weights and their desired values, and the construction of transport mappings via Fourier transforms. Regarding the first contribution, the proof …
CSAIL, Massachusetts Institute of Technology（麻省理工学院计算机科学与人工智能实验室）
Current graph neural network (GNN) architectures naively average or sum node embeddings into an aggregated graph representation -- potentially losing structural or semantic information. We here introduce OT-GNN, a model that computes graph embeddings using parametric prototypes that highlight key facets of different graph aspects. Towards this goal, we successfully combine optimal transport (OT) with parametric graph models. Graph representations are obtained from Wasserstein distances between the set of GNN node embeddings and ``prototype'' point clouds as free parameters. We theoretically pr…
Bayesian Optimisation (BO) refers to a class of methods for global optimisation of a function $f$ which is only accessible via point evaluations. It is typically used in settings where $f$ is expensive to evaluate. A common use case for BO in machine learning is model selection, where it is not possible to analytically model the generalisation performance of a statistical model, and we resort to noisy and expensive training and validation procedures to choose the best model. Conventional BO methods have focused on Euclidean and categorical domains, which, in the context of model selection, onl…
We investigated the feature map inside deep neural networks (DNNs) by tracking the transport map. We are interested in the role of depth (why do DNNs perform better than shallow models?) and the interpretation of DNNs (what do intermediate layers do?) Despite the rapid development in their application, DNNs remain analytically unexplained because the hidden layers are nested and the parameters are not faithful. Inspired by the integral representation of shallow NNs, which is the continuum limit of the width, or the hidden unit number, we developed the flow representation and transport analysis…
Amazon（亚马逊）；RIKEN AIP（理化学研究所先进智能项目中心）；Kyoto University（京都大学）；University of Oxford（牛津大学）
Neural architecture search (NAS) automates the design of deep neural networks. One of the main challenges in searching complex and non-continuous architectures is to compare the similarity of networks that the conventional Euclidean metric may fail to capture. Optimal transport (OT) is resilient to such complex structure by considering the minimal cost for transporting a network into another. However, the OT is generally not negative definite which may limit its ability to build the positive-definite kernels required in many kernel-dependent frameworks. Building upon tree-Wasserstein (TW), whi…
Columbia University（哥伦比亚大学）；University of Amsterdam（阿姆斯特丹大学）
Variational inference often minimizes the "reverse" Kullbeck-Leibler (KL) KL(q||p) from the approximate distribution q to the posterior p. Recent work studies the "forward" KL KL(p||q), which unlike reverse KL does not lead to variational approximations that underestimate uncertainty. This paper introduces Transport Score Climbing (TSC), a method that optimizes KL(p||q) by using Hamiltonian Monte Carlo (HMC) and a novel adaptive transport map. The transport map improves the trajectory of HMC by acting as a change of variable between the latent variable space and a warped space. TSC uses HMC sa…
