We study an optimization problem related to the approximation of given data by a linear combination of transformed modes. In the simplest case, the optimization problem reduces to a minimization problem well-studied in the context of proper orthogonal decomposition. Allowing transformed modes in the approximation renders this approach particularly useful to compress data with transported quantities, which are prevalent in many flow applications. We prove the existence of a solution to the infinite-dimensional optimization problem. Towards a numerical implementation, we compute the gradient of the cost functional and derive a suitable discretization in time and space. We demonstrate the theoretical findings with three challenging numerical examples.
We study Benamou's domain decomposition algorithm for optimal transport in the entropy regularized setting. The key observation is that the regularized variant converges to the globally optimal solution under very mild assumptions. We prove linear convergence of the algorithm with respect to the Kullback--Leibler divergence and illustrate the (potentially very slow) rates with numerical examples. On problems with sufficient geometric structure (such as Wasserstein distances between images) we expect much faster convergence. We then discuss important aspects of a computationally efficient imple…
Proximal Gradient-based Low Rank Tensor Decomposition for State Dependent Riccati Equation
Institute of Mathematical Science, ShanghaiTech University（上海科技大学数学科学学院）；Department of Mathematics, University of Alabama at Birmingham（阿拉巴马大学伯明翰分校数学系）
Large optimal transport problems can be approached via domain decomposition, i.e. by iteratively solving small partial problems independently and in parallel. Convergence to the global minimizers under suitable assumptions has been shown in the unregularized and entropy regularized setting and its computational efficiency has been demonstrated experimentally. An accurate theoretical understanding of its convergence speed in geometric settings is still lacking. In this article we work towards such an understanding by deriving, via $Γ$-convergence, an asymptotic description of the algorithm in t…
A new decomposition optimization algorithm, called \textit{path-following gradient-based decomposition}, is proposed to solve separable convex optimization problems. Unlike path-following Newton methods considered in the literature, this algorithm does not requires any smoothness assumption on the objective function. This allows us to handle more general classes of problems arising in many real applications than in the path-following Newton methods. The new algorithm is a combination of three techniques, namely smoothing, Lagrangian decomposition and path-following gradient framework. The algo…
Microsoft Research New England（微软新英格兰研究院）；University of California Irvine（加州大学欧文分校）；University of California Berkeley（加州大学伯克利分校）；Cornell University（康奈尔大学）
We analyze stochastic gradient descent for optimizing non-convex functions. In many cases for non-convex functions the goal is to find a reasonable local minimum, and the main concern is that gradient updates are trapped in saddle points. In this paper we identify strict saddle property for non-convex problem that allows for efficient optimization. Using this property we show that stochastic gradient descent converges to a local minimum in a polynomial number of iterations. To the best of our knowledge this is the first work that gives global convergence guarantees for stochastic gradient desc…
None of the BROJA information decomposition measures $\mbox{SI}, \mbox{CI}, \mbox{UIy}, \mbox{UIz}$ are convex or concave over the probability simplex. In this paper, we provide formulas for the sub-gradient and super-gradients of any of the information decomposition measures. Then we apply these results to obtain an optimum of some of these information decomposition measures when optimized over a constrained set of probability distributions.
We study the consequences of the equivalence between the least gradient problem and a boundary-to-boundary optimal transport problem in two dimensions. We extend the relationship between the two problems to their respective dual problems, as well as prove several regularity and stability results for the least gradient problem using optimal transport techniques.
In this paper, a green transportation location problem is considered with uncertain demand parameter. Increasing robustness influences the number of trucks for sending goods and products, and consequently, makes the air pollution enhance. In this paper, two green approaches are introduced which demand is the main uncertain parameter in both. These approaches are addressed to provide a trade-off between using available trucks and buying new hybrid trucks for evaluating total costs besides air pollution. Due to growing complexity, a Lagrangian decomposition algorithm is applied to find a tight l…
Assessing non-negativity of multivariate polynomials over the reals, through the computation of {\em certificates of non-negativity}, is a topical issue in polynomial optimization. This is usually tackled through the computation of {\em sums-of-squares decompositions} which rely on efficient numerical solvers for semi-definite programming. This method faces two difficulties. The first one is that the certificates obtained this way are {\em approximate} and then non-exact. The second one is due to the fact that not all non-negative polynomials are sums-of-squares. In this paper, we build on pre…
We study a natural Wasserstein gradient flow on manifolds of probability distributions with discrete sample spaces. We derive the Riemannian structure for the probability simplex from the dynamical formulation of the Wasserstein distance on a weighted graph. We pull back the geometric structure to the parameter space of any given probability model, which allows us to define a natural gradient flow there. In contrast to the natural Fisher-Rao gradient, the natural Wasserstein gradient incorporates a ground metric on sample space. We illustrate the analysis of elementary exponential family examp…
In the setting of optimal transport with $N\ge 2$ marginals, a necessary condition for transport plans to be optimal is that they are $c$-cyclically monotone. For $N=2$ there exist several proofs that in very general settings $c$-cyclical monotoncity is also sufficient for optimality, while for $N\ge 3$ this is only known under strong conditions on $c$. Here we give a counterexample which shows that $c$-cylclical monotonicity is in general not sufficient for optimality if $N\ge 3$. Comparison with the $N=2$ case shows how the main proof strategies valid for the case $N=2$ might fail for $N\ge …
Center for Data Science（数据科学中心）；Peking University（北京大学）；Key Lab. of Machine Perception (MOE)（机器感知教育部重点实验室）；School of EECS, Peking University（北京大学电子工程与计算机科学学院）；Peng Cheng Laboratory（鹏城实验室）
Adaptive gradient methods, especially Adam-type methods (such as Adam, AMSGrad, and AdaBound), have been proposed to speed up the training process with an element-wise scaling term on learning rates. However, they often generalize poorly compared with stochastic gradient descent (SGD) and its accelerated schemes such as SGD with momentum (SGDM). In this paper, we propose a new adaptive method called DecGD, which simultaneously achieves good generalization like SGDM and obtain rapid convergence like Adam-type methods. In particular, DecGD decomposes the current gradient into the product of two …
Coherent ptychographic imaging experiments often discard over 99.9 % of the flux from a light source to define the coherence of an illumination. Even when coherent flux is sufficient, the stability required during an exposure is another important limiting factor. Partial coherence analysis can considerably reduce these limitations. A partially coherent illumination can often be written as the superposition of a single coherent illumination convolved with a separable translational kernel. In this paper we propose the Gradient Decomposition of the Probe (GDP), a model that exploits translational…
Toyota Technological Institute at Chicago（芝加哥丰田技术研究所）
Recently, several works have shown that natural modifications of the classical conditional gradient method (aka Frank-Wolfe algorithm) for constrained convex optimization, provably converge with a linear rate when: i) the feasible set is a polytope, and ii) the objective is smooth and strongly-convex. However, all of these results suffer from two significant shortcomings: large memory requirement due to the need to store an explicit convex decomposition of the current iterate, and as a consequence, large running-time overhead per iteration, and worst case convergence rate that depends unfavora…
This article deals with the conjugate gradient method on a Riemannian manifold with interest in global convergence analysis. The existing conjugate gradient algorithms on a manifold endowed with a vector transport need the assumption that the vector transport does not increase the norm of tangent vectors, in order to confirm that generated sequences have a global convergence property. In this article, the notion of a scaled vector transport is introduced to improve the algorithm so that the generated sequences may have a global convergence property under a relaxed assumption. In the proposed a…
The least gradient problem (minimizing the total variation with given boundary data) is equivalent, in the plane, to the Beckmann minimal-flow problem with source and target measures located on the boundary of the domain, which is in turn related to an optimal transport problem. Motivated by this fact, we prove L p summability results for the solution of the Beckmann problem in this setting, which improve upon previous results where the measures were themselves supposed to be L p. In the plane, we carry out all the analysis for general strictly convex norms, which requires to first introduce t…
The papers describes an easy approach, based on a classical construction by Dacorogna and Moser, to prove that optimal vector fields in some minimal flow problem linked to optimal transport models (congested traffic, branched transport, Beckmann's problem...) are induced by a probability measure on the space of paths. This gives a new, easier, proof of a classical result by Smirnov, and allows handling optimal flows without taking care of the presence of cycles.
University of Southern California（南加州大学）；Google Brain（谷歌大脑）；University of British Columbia（不列颠哥伦比亚大学）；Mila（米拉研究所）；Université de Montréal（蒙特利尔大学）
Most stochastic optimization methods use gradients once before discarding them. While variance reduction methods have shown that reusing past gradients can be beneficial when there is a finite number of datapoints, they do not easily extend to the online setting. One issue is the staleness due to using past gradients. We propose to correct this staleness using the idea of implicit gradient transport (IGT) which transforms gradients computed at previous iterates into gradients evaluated at the current iterate without using the Hessian explicitly. In addition to reducing the variance and bias of…
Many tasks in machine learning and signal processing can be solved by minimizing a convex function of a measure. This includes sparse spikes deconvolution or training a neural network with a single hidden layer. For these problems, we study a simple minimization method: the unknown measure is discretized into a mixture of particles and a continuous-time gradient descent is performed on their weights and positions. This is an idealization of the usual way to train neural networks with a large hidden layer. We show that, when initialized correctly and in the many-particle limit, this gradient fl…
