Estimating optical flows is one of the most interesting problems in computer vision, which estimates the essential information about pixel-wise displacements between two consecutive images. This work introduces an efficient dual optimization framework with accelerated preconditioners to the challenging nonsmooth optimization problem of total-variation regularized optical-flow estimation. In theory, the proposed dual optimization framework brings an elegant variational analysis on the given difficult optimization problem, while presenting an efficient algorithmic scheme without directly tackling the corresponding nonsmoothness in numeric. By introducing efficient preconditioners with a multi-scale implementation, the proposed accelerated dual optimization approaches achieve competitive estimation results of image motion, comparing to the state-of-the-art methods. Moreover, we show that the proposed preconditioners can guarantee convergence of the implemented numerical schemes with high efficiency.
A variational formulation of accelerated optimization on normed spaces was recently introduced by considering a specific family of time-dependent Bregman Lagrangian and Hamiltonian systems whose corresponding trajectories converge to the minimizer of the given convex function at an arbitrary accelerated rate of $\mathcal{O}(1/t^p)$. This framework has been exploited using time-adaptive geometric integrators to design efficient explicit algorithms for symplectic accelerated optimization. It was observed that geometric discretizations were substantially less prone to stability issues, and were t…
There has been significant interest in generalizations of the Nesterov accelerated gradient descent algorithm due to its improved performance guarantee compared to the standard gradient descent algorithm, and its applicability to large scale optimization problems arising in deep learning. A particularly fruitful approach is based on numerical discretizations of differential equations that describe the continuous time limit of the Nesterov algorithm, and a generalization involving time-dependent Bregman Lagrangian and Hamiltonian dynamics that converges at an arbitrarily fast rate to the minimu…
The continuous-time model of Nesterov's momentum provides a thought-provoking perspective for understanding the nature of the acceleration phenomenon in convex optimization. One of the main ideas in this line of research comes from the field of classical mechanics and proposes to link Nesterov's trajectory to the solution of a set of Euler-Lagrange equations relative to the so-called Bregman Lagrangian. In the last years, this approach led to the discovery of many new (stochastic) accelerated algorithms and provided a solid theoretical foundation for the design of structure-preserving accelera…
We propose a novel method, namely the accelerated mirror-prox (AMP) method, for computing the weak solutions of a class of deterministic and stochastic monotone variational inequalities (VI). The main idea of this algorithm is to incorporate a multi-step acceleration scheme into the mirror-prox method. For both deterministic and stochastic VIs, the developed AMP method computes the weak solutions with optimal rate of convergence. In particular, if the monotone operator in VI consists of the gradient of a smooth function, the rate of convergence of the AMP method can be accelerated in terms of …
This paper investigates two accelerated primal-dual mirror dynamical approaches for smooth and nonsmooth convex optimization problems with affine and closed, convex set constraints. In the smooth case, an accelerated primal-dual mirror dynamical approach (APDMD) based on accelerated mirror descent and primal-dual framework is proposed and accelerated convergence properties of primal-dual gap, feasibility measure and the objective function value along with trajectories of APDMD are derived by the Lyapunov analysis method. Then, we extend APDMD into two distributed dynamical approaches to deal w…
It was shown recently by Su et al. (2016) that Nesterov's accelerated gradient method for minimizing a smooth convex function $f$ can be thought of as the time discretization of a second-order ODE, and that $f(x(t))$ converges to its optimal value at a rate of $\mathcal{O}(1/t^2)$ along any trajectory $x(t)$ of this ODE. A variational formulation was introduced in Wibisono et al. (2016) which allowed for accelerated convergence at a rate of $\mathcal{O}(1/t^p)$, for arbitrary $p>0$, in normed vector spaces. This framework was exploited in Duruisseaux et al. (2021) to design efficient explicit …
This work proposes an accelerated primal-dual dynamical system for affine constrained convex optimization and presents a class of primal-dual methods with nonergodic convergence rates. In continuous level, exponential decay of a novel Lyapunov function is established and in discrete level, implicit, semi-implicit and explicit numerical discretizations for the continuous model are considered sequentially and lead to new accelerated primal-dual methods for solving linearly constrained optimization problems. Special structures of the subproblems in those schemes are utilized to develop efficient …
In this technical note, we are concerned with the problem of solving variational inequalities with improved convergence rates. Motivated by Nesterov's accelerated gradient method for convex optimization, we propose a Nesterov's accelerated projected gradient algorithm for variational inequality problems. We prove convergence of the proposed algorithm with at least linear rate under the common assumption of Lipschitz continuity and strongly monotonicity. To the best of our knowledge, this is the first time that convergence of the Nesterov's accelerated protocol is proved for variational inequal…
This paper develops a continuous-time primal-dual accelerated method with an increasing damping coefficient for a class of convex optimization problems with affine equality constraints. This paper analyzes critical values for parameters in the proposed method and prove that the rate of convergence in terms of the duality gap function is $O(\tfrac{1}{t^2})$ by choosing suitable parameters. As far as we know, this is the first continuous-time primal-dual accelerated method that can obtain the optimal rate. Then this work applies the proposed method to two network optimization problems, a distrib…
In this paper, we propose a unifying framework incorporating several momentum-related search directions for solving strongly monotone variational inequalities. The specific combinations of the search directions in the framework are made to guarantee the optimal iteration complexity bound of $\mathcal{O}\left(κ\ln(1/ε)\right)$ to reach an $ε$-solution, where $κ$ is the condition number. This framework provides the flexibility for algorithm designers to train -- among different parameter combinations -- the one that best suits the structure of the problem class at hand. The proposed framework in…
This work presents a universal accelerated first-order primal-dual method for affinely constrained convex optimization problems. It can handle both Lipschitz and Hölder gradients but does not need to know the smoothness level of the objective function. In line search part, it uses dynamically decreasing parameters and produces approximate Lipschitz constant with moderate magnitude. In addition, based on a suitable discrete Lyapunov function and tight decay estimates of some differential/difference inequalities, a universal optimal mixed-type convergence rate is established. Some numerical test…
This paper presents a simple primal dual method named DPD which is a flexible framework for a class of saddle point problem with or without strongly convex component. The presented method has linearized version named LDPD and exact version EDPD. Each iteration of DPD updates sequentially the dual and primal variable via simple proximal mapping and refines the dual variable via extrapolation. Convergence analysis with smooth or strongly convex primal component recovers previous state-of-the-art results, and that with strongly convex dual component attains full acceleration $O(1/k^2)$ in terms o…
In this paper, a new variant of accelerated gradient descent is proposed. The pro-posed method does not require any information about the objective function, usesexact line search for the practical accelerations of convergence, converges accordingto the well-known lower bounds for both convex and non-convex objective functions,possesses primal-dual properties and can be applied in the non-euclidian set-up. Asfar as we know this is the rst such method possessing all of the above properties atthe same time. We also present a universal version of the method which is applicableto non-smooth proble…
We introduce primal and dual stochastic gradient oracle methods for decentralized convex optimization problems. Both for primal and dual oracles, the proposed methods are optimal in terms of the number of communication steps. However, for all classes of the objective, the optimality in terms of the number of oracle calls per node takes place only up to a logarithmic factor and the notion of smoothness. By using mini-batching technique, we show that the proposed methods with stochastic oracle can be additionally parallelized at each node. The considered algorithms can be applied to many data sc…
This article develops a duality principle applicable to a large class of variational problems. Firstly, we apply the results to a Ginzburg-Landau type model. In a second step, we develop another duality principle and related primal dual variational formulation and such an approach includes optimality conditions which guarantee zero duality gap between the primal and dual formulations. We emphasize in both cases the dual variational formulations obtained have large regions of concavity about the critical points in question.
A mixed type dual to a nondifferentiable variational problem involving higher order derivative is formulated and duality results are proved under generalized invexity conditions. Special cases are generated from our results.
This paper investigates accelerating the convergence of distributed optimization algorithms on non-convex problems. We propose a distributed primal-dual stochastic gradient descent~(SGD) equipped with "powerball" method to accelerate. We show that the proposed algorithm achieves the linear speedup convergence rate $\mathcal{O}(1/\sqrt{nT})$ for general smooth (possibly non-convex) cost functions. We demonstrate the efficiency of the algorithm through numerical experiments by training two-layer fully connected neural networks and convolutional neural networks on the MNIST dataset to compare wit…
This thesis focuses on developing and analyzing accelerated and inexact first-order methods for solving or finding stationary points of various nonconvex composite optimization (NCO) problems. The main tools mainly come from variational and convex analysis, and the key results are in the form of iteration complexity bounds and how these bounds compare to other ones in the literature.
Accelerated gradient methods play a central role in optimization, achieving optimal rates in many settings. While many generalizations and extensions of Nesterov's original acceleration method have been proposed, it is not yet clear what is the natural scope of the acceleration concept. In this paper, we study accelerated methods from a continuous-time perspective. We show that there is a Lagrangian functional that we call the \emph{Bregman Lagrangian} which generates a large class of accelerated methods in continuous time, including (but not limited to) accelerated gradient descent, its non-E…
This short communication develops a convex dual variational formulation for a large class of models in variational optimization. The results are established through basic tools of functional analysis, convex analysis and duality theory. The main duality principle is developed as an application to a Ginzburg-Landau type system in superconductivity in the absence of a magnetic field.
In this paper, we propose a variance-reduced primal-dual algorithm with Bregman distance for solving convex-concave saddle-point problems with finite-sum structure and nonbilinear coupling function. This type of problems typically arises in machine learning and game theory. Based on some standard assumptions, the algorithm is proved to converge with oracle complexity of $O\left(\frac{\sqrt n}ε\right)$ and $O\left(\frac{n}{\sqrt ε}+\frac{1}{ε^{1.5}}\right)$ using constant and non-constant parameters, respectively where $n$ is the number of function components. Compared with existing methods, ou…
We consider a distributed optimization problem over a network of agents aiming to minimize a global objective function that is the sum of local convex and composite cost functions. To this end, we propose a distributed Chebyshev-accelerated primal-dual algorithm to achieve faster ergodic convergence rates. In standard distributed primal-dual algorithms, the speed of convergence towards a global optimum (i.e., a saddle point in the corresponding Lagrangian function) is directly influenced by the eigenvalues of the Laplacian matrix representing the communication graph. In this paper, we use Cheb…
This article develops dual variational formulations for a large class of models in variational optimization. The results are established through basic tools of functional analysis, convex analysis and duality theory. The main duality principle is developed as an application to a Ginzburg-Landau type system in superconductivity in the absence of a magnetic field. In the first part final sections, we develop new general dual convex variational formulations, more specifically, dual formulations with a large region of convexity around the critical points which are suitable for the non-convex optim…
This article develops duality principles applicable to the non-linear Kirchhoff-Love model of plates. The results are obtained through standard tools of convex analysis, functional analysis, calculus of variations and duality theory. The main duality principle concerns a convex (in fact concave) dual variational formulation and related new optimality conditions for the model in question. Finally, in the last section we develop some global existence results for a similar model in elasticity.
In this article we develop a new primal dual variational formulation suitable for a large class of non-convex problems in the calculus of variations. The results are obtained through basic tools of convex analysis, duality theory, the Legendre transform concept and the respective relations between the primal and dual variables. The novelty here is that the dual formulation is established also for the primal variables, however with a large domain region of concavity about a critical point. Finally, we formally prove there is no duality gap between the primal and dual formulations in a local ext…
This paper discusses the efficiency of Hybrid Primal-Dual (HPD) type algorithms to approximate solve discrete Optimal Transport (OT) and Wasserstein Barycenter (WB) problems, with and without entropic regularization. Our first contribution is an analysis showing that these methods yield state-of-the-art convergence rates, both theoretically and practically. Next, we extend the HPD algorithm with linesearch proposed by Malitsky and Pock in 2018 to the setting where the dual space has a Bregman divergence, and the dual function is relatively strongly convex to the Bregman's kernel. This extensio…
This article develops a new primal dual formulation for the Kirchhoff-Love non-linear plate model. At first we establish a duality principle which includes sufficient conditions of global optimality through the dual formulation. At this point we highlight this first duality principle is specially suitable for the case in which the membrane stress tensor is negative definite. In a second step, from such a general principle, we develop a primal dual variational formulation which also includes the corresponding sufficient conditions for global optimality. The results are based on standard tools o…
We develop an Accelerated Back Pressure (ABP) algorithm using Accelerated Dual Descent (ADD), a distributed approximate Newton-like algorithm that only uses local information. Our construction is based on writing the backpressure algorithm as the solution to a network feasibility problem solved via stochastic dual subgradient descent. We apply stochastic ADD in place of the stochastic gradient descent algorithm. We prove that the ABP algorithm guarantees stable queues. Our numerical experiments demonstrate a significant improvement in convergence rate, especially when the packet arrival statis…
This article develops a primal dual formulation for a primal proximal approach suitable for a large class of non-convex models in the calculus of variations. The results are established through standard tools of functional analysis, convex analysis and duality theory and are applied to a Ginzburg-Landau type model. Finally, in the last two sections, we present concerning optimal conditions and another related duality principle for the model in question.
This paper proposes an adaptive primal-dual dynamics for distributed optimization in multi-agent systems. The proposed dynamics incorporates an adaptive synchronization law that reinforces the interconnection strength between the primal variables of the coupled agents, the given law accelerates the convergence of the proposed dynamics to the saddle-point solution. The resulting dynamics is represented as a feedback interconnected networked system that proves to be passive. The passivity properties of the proposed dynamics are exploited along with the LaSalle's invariance principle for hybrid s…
This is a short summary of already published results on accelerated first and zero-order optimization methods, as well as accelerated methods for problems with linear constraints. This short summary is a requirement for obtaining a degree of doctor of sciences in Russian Federation. The details can be found in the papers listed in the introduction.
We consider a multimarginal optimal transport, which includes as a particular case the Wasserstein barycenter problem. In this problem one has to find an optimal coupling between $m$ probability measures, which amounts to finding a tensor of the order $m$. We propose an accelerated method based on accelerated alternating minimization and estimate its complexity to find the approximate solution to the problem. We use entropic regularization with sufficiently small regularization parameter and apply accelerated alternating minimization to the dual problem. A novel primal-dual analysis is used to…
This work presents an algorithmic scheme for solving the infinite-time constrained linear quadratic regulation problem. We employ an accelerated version of a popular proximal gradient scheme, commonly known as the Forward-Backward Splitting (FBS), and prove its convergence to the optimal solution in our infinite-dimensional setting. Each iteration of the algorithm requires only finite memory, is computationally cheap, and makes no use of terminal invariant sets; hence, the algorithm can be applied to systems of very large dimensions. The acceleration brings in optimal convergence rates O(1/k^2…
This article develops duality principles applicable to non-convex models in the calculus of variations. The results here developed are applied to Ginzburg-Landau type equations. For the first and second duality principles, through an optimality criterion developed for the dual formulations, we qualitatively classify the critical points of the primal and dual functionals in question. We formally prove there is no duality gap between the primal and dual formulations in a local extremal context. Finally, in the last sections, we present a global existence result, a duality principle and respectiv…
WeBank AI Group（微众银行AI集团）；Tencent Inc.（腾讯公司）；University of California, Davis（加州大学戴维斯分校）；The Hong Kong University of Science and Technology（香港科技大学）
Dual averaging-type methods are widely used in industrial machine learning applications due to their ability to promoting solution structure (e.g., sparsity) efficiently. In this paper, we propose a novel accelerated dual-averaging primal-dual algorithm for minimizing a composite convex function. We also derive a stochastic version of the proposed method which solves empirical risk minimization, and its advantages on handling sparse data are demonstrated both theoretically and empirically.
An accelerated block coordinate descent (ABCD) method in Hilbert space is analyzed to solve the sparse optimal control problem via its dual. The finite element approximation of this method is investigated and convergence results are presents. Based on the second order growth condition of the dual objective function, we show that iteration sequence of dual variables has the iteration complexity of $O(1/k)$. Moreover, we also prove iteration complexity for the primal problem. Two types of mesh-independence for ABCD method are proved, which asserts that asymptotically the infinite dimensional ABC…
We show how one can obtain nonaccelerated randomized coordinate descent method (Yu. Nesterov, 2010) and nonaccelerated method of randomization of sum-type functional (Le Roux-Schmidt-Bach, 2012) from the optimal method for the stochastic optimization problem (SIGMA, Devolder-Glineur-Nesterov-Dvurechensky-Gasnikov, 2014). The main trick is a special restart technique. We considered this trick to be usefull in others contexts. We consider only strongly convex case. We show that accelerated variants of this methods seems to be nontrivial in this context. That is, it is hard (perhaps impossible) t…
