Flow-based models are powerful tools for designing probabilistic models with tractable density. This paper introduces Convex Potential Flows (CP-Flow), a natural and efficient parameterization of invertible models inspired by the optimal transport (OT) theory. CP-Flows are the gradient map of a strongly convex neural potential function. The convexity implies invertibility and allows us to resort to convex optimization to solve the convex conjugate for efficient inversion. To enable maximum likelihood training, we derive a new gradient estimator of the log-determinant of the Jacobian, which involves solving an inverse-Hessian vector product using the conjugate gradient method. The gradient estimator has constant-memory cost, and can be made effectively unbiased by reducing the error tolerance level of the convex optimization routine. Theoretically, we prove that CP-Flows are universal density approximators and are optimal in the OT sense. Our empirical results show that CP-Flow performs competitively on standard benchmarks of density estimation and variational inference.
University of Michigan（密歇根大学）；Princeton University（普林斯顿大学）
This paper explores a surprising equivalence between two seemingly-distinct convex optimization methods. We show that simulated annealing, a well-studied random walk algorithms, is directly equivalent, in a certain sense, to the central path interior point algorithm for the the entropic universal barrier function. This connection exhibits several benefits. First, we are able improve the state of the art time complexity for convex optimization under the membership oracle model. We improve the analysis of the randomized algorithm of Kalai and Vempala by utilizing tools developed by Nesterov and …
National Key Laboratory for Novel Software Technology, Nanjing University（南京大学计算机软件新技术国家重点实验室）
In this paper, we study adaptive online convex optimization, and aim to design a universal algorithm that achieves optimal regret bounds for multiple common types of loss functions. Existing universal methods are limited in the sense that they are optimal for only a subclass of loss functions. To address this limitation, we propose a novel online method, namely Maler, which enjoys the optimal $O(\sqrt{T})$, $O(d\log T)$ and $O(\log T)$ regret bounds for general convex, exponentially concave, and strongly convex functions respectively. The essential idea is to run multiple types of learning alg…
National Key Laboratory for Novel Software Technology（现代软件工程国家重点实验室）；School of Artificial Intelligence, Nanjing University（南京大学人工智能学院）；College of Computing, Georgia Institute of Technology（佐治亚理工学院计算学院）；JD AI Research（京东AI研究院）
Universal Online Convex Optimization with $1$ Projection per Round
National Key Laboratory for Novel Software Technology, Nanjing University（南京大学计算机软件新技术国家重点实验室）；School of Artificial Intelligence, Nanjing University（南京大学人工智能学院）
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of California, Irvine（加州大学欧文分校）；Princeton University（普林斯顿大学）；University of Washington（华盛顿大学）
In this paper, we present a novel and principled approach to learn the optimal transport between two distributions, from samples. Guided by the optimal transport theory, we learn the optimal Kantorovich potential which induces the optimal transport map. This involves learning two convex functions, by solving a novel minimax optimization. Building upon recent advances in the field of input convex neural networks, we propose a new framework where the gradient of one convex function represents the optimal transport mapping. Numerical experiments confirm that we learn the optimal transport mapping…
National Key Laboratory for Novel Software Technology, Nanjing University（南京大学新型软件技术国家重点实验室）；Paradigm Inc.（第四范式有限公司）
To deal with changing environments, a new performance measure -- adaptive regret, defined as the maximum static regret over any interval, was proposed in online learning. Under the setting of online convex optimization, several algorithms have been successfully developed to minimize the adaptive regret. However, existing algorithms lack universality in the sense that they can only handle one type of convex functions and need apriori knowledge of parameters. By contrast, there exist universal algorithms, such as MetaGrad, that attain optimal static regret for multiple types of convex functions …
Optimal transport (OT) theory underlies many emerging machine learning (ML) methods nowadays solving a wide range of tasks such as generative modeling, transfer learning and information retrieval. These latter works, however, usually build upon a traditional OT setup with two distributions, while leaving a more general multi-marginal OT formulation somewhat unexplored. In this paper, we study the multi-marginal OT (MMOT) problem and unify several popular OT methods under its umbrella by promoting structural information on the coupling. We show that incorporating such structural information int…
Dual Adaptivity: Universal Algorithms for Minimizing the Adaptive Regret of Convex Functions
National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China（南京大学新型软件技术国家重点实验室）；School of Artificial Intelligence, Nanjing University, Nanjing, China（南京大学人工智能学院）；College of Computing, Georgia Institute of Technology, Atlanta, USA（佐治亚理工学院计算机学院）
University of Utah（犹他大学）；Florida State University（佛罗里达州立大学）
Optimal Mass Transport (OMT) is a well studied problem with a variety of applications in a diverse set of fields ranging from Physics to Computer Vision and in particular Statistics and Data Science. Since the original formulation of Monge in 1781 significant theoretical progress been made on the existence, uniqueness and properties of the optimal transport maps. The actual numerical computation of the transport maps, particularly in high dimensions, remains a challenging problem. By Brenier's theorem, the continuous OMT problem can be reduced to that of solving a non-linear PDE of Monge-Amper…
Recently, several universal methods have been proposed for online convex optimization which can handle convex, strongly convex and exponentially concave cost functions simultaneously. However, most of these algorithms have been designed with static regret minimization in mind, but this notion of regret may not be suitable for changing environments. To address this shortcoming, we propose a novel and intuitive framework for universal online optimization in dynamic environments. Unlike existing universal algorithms, our strategy does not rely on the construction of a set of experts and an accomp…
Generative modelling is a key tool in unsupervised machine learning which has achieved stellar success in recent years. Despite this huge success, even the best generative models such as Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs) come with their own shortcomings, mode collapse and mode mixture being the two most prominent problems. In this paper we develop a new generative model capable of generating samples which resemble the observed data, and is free from mode collapse and mode mixture. Our model is inspired by the recently proposed Autoencoder-Optimal Transp…
Seoul National University（首尔国立大学）；Institute of Advanced Aerospace Technology（先进航空航天技术研究所）
Parameterized max-affine (PMA) and parameterized log-sum-exp (PLSE) networks are proposed for general decision-making problems. The proposed approximators generalize existing convex approximators, namely, max-affine (MA) and log-sum-exp (LSE) networks, by considering function arguments of condition and decision variables and replacing the network parameters of MA and LSE networks with continuous functions with respect to the condition variable. The universal approximation theorem of PMA and PLSE is proven, which implies that PMA and PLSE are shape-preserving universal approximators for paramet…
INRIA（法国国家信息与自动化研究所）；ENS（巴黎高等师范学院）；PSL Research University（巴黎文理研究大学）；ETH Zürich（苏黎世联邦理工学院）
We consider variational inequalities coming from monotone operators, a setting that includes convex minimization and convex-concave saddle-point problems. We assume an access to potentially noisy unbiased values of the monotone operators and assess convergence through a compatible gap function which corresponds to the standard optimality criteria in the aforementioned subcases. We present a universal algorithm for these inequalities based on the Mirror-Prox algorithm. Concretely, our algorithm simultaneously achieves the optimal rates for the smooth/non-smooth, and noisy/noiseless settings. Th…
Convex Physics Informed Neural Networks for the Monge-Ampère Optimal Transport Problem
Geneva School of Business Administration (HEG), University of Applied Sciences；Institute of Mathematics, Ecole Polytechnique F\'ed\'erale de Lausanne, 1015 Lausanne, Switzerland, Email
