The derivation of second-order ordinary differential equations (ODEs) as continuous-time limits of optimization algorithms has been shown to be an effective tool for the analysis of these algorithms. Additionally, discretizing generalizations of these ODEs can lead to new families of optimization methods. We study discretizations of an Euler-Lagrange equation which generate a large class of accelerated methods whose convergence rate is $O(\frac{1}{t^p})$ in continuous-time, where parameter $p$ is the order of the optimization method. Specifically, we address the question asking why a naive explicit-implicit Euler discretization of this solution produces an unstable algorithm, even for a strongly convex objective function. We prove that for a strongly convex $L$-smooth quadratic objective function and step size $δ<\frac{1}{L}$, the naive discretization will exhibit stable behavior when the number of iterations $k$ satisfies the inequality $k < (\frac{4}{Lp^2 δ^p})^{\frac{1}{p-2}}$. Additionally, we extend our analysis to the implicit and explicit Euler discretization methods to determine end behavior.
We study gradient-based optimization methods obtained by direct Runge-Kutta discretization of the ordinary differential equation (ODE) describing the movement of a heavy-ball under constant friction coefficient. When the function is high order smooth and strongly convex, we show that directly simulating the ODE with known numerical integrators achieve acceleration in a nontrivial neighborhood of the optimal solution. In particular, the neighborhood can grow larger as the condition number of the function increases. Furthermore, our results also hold for nonconvex but quasi-strongly convex objec…
We develop a distributed algorithm for convex Empirical Risk Minimization, the problem of minimizing large but finite sum of convex functions over networks. The proposed algorithm is derived from directly discretizing the second-order heavy-ball differential equation and results in an accelerated convergence rate, i.e, faster than distributed gradient descent-based methods for strongly convex objectives that may not be smooth. Notably, we achieve acceleration without resorting to the well-known Nesterov's momentum approach. We provide numerical experiments and contrast the proposed method with…
Discretization methods for differential-algebraic equations (DAEs) are considered that are based on the integration of an associated inherent ordinary differential equation (ODE). This allows to make use of any discretization scheme suitable for the numerical integration of ODEs. For DAEs with symmetries it is shown that the inherent ODE can be constructed in such a way that it inherits the symmetry properties of the given DAE and geometric properties of its flow. This in particular allows the use of geometric integration schemes with a numerical flow that has analogous geometric properties.
In this paper we investigate a priori error estimates for the space-time Galerkin finite element discretization of an optimal control problem governed by a simplified linear gradient enhanced damage model. The model equations are of a special structure as the state equation consists of an elliptic PDE which has to be fulfilled at almost all times coupled with an ODE that has to hold true in almost all points in space. The state equation is discretized by a piecewise constant discontinuous Galerkin method in time and usual conforming linear finite elements in space. For the discretization of th…
In this paper we investigate a priori error estimates for the space-time Galerkin finite element discretization of a simplified semilinear gradient enhanced damage model. The model equations are of a special structure as the state equation consists of an elliptic PDE which has to be fulfilled at almost all times coupled with a non-smooth, semilinear ODE that has to hold true in almost all points in space. The system is discretized by a constant discontinuous Galerkin method in time and usual conforming linear finite elements in space. For the uncontrolled equation, we prove linear convergence …
University of California, Berkeley（加州大学伯克利分校）；Institute for Advanced Study（普林斯顿高等研究院）；University of Pennsylvania（宾夕法尼亚大学）
We study first-order optimization methods obtained by discretizing ordinary differential equations (ODEs) corresponding to Nesterov's accelerated gradient methods (NAGs) and Polyak's heavy-ball method. We consider three discretization schemes: an explicit Euler scheme, an implicit Euler scheme, and a symplectic scheme. We show that the optimization algorithm generated by applying the symplectic scheme to a high-resolution ODE proposed by Shi et al. [2018] achieves an accelerated rate for minimizing smooth strongly convex functions. On the other hand, the resulting algorithm either fails to ach…
In this work, we present a scalable Linear Matrix Inequality (LMI) based framework to verify the stability of a set of linear Partial Differential Equations (PDEs) in one spatial dimension coupled with a set of Ordinary Differential Equations (ODEs) via input-output based interconnection. Our approach extends the newly developed state space representation and stability analysis of coupled PDEs that allows parametrizing the Lyapunov function on $L_2$ with multipliers and integral operators using polynomial kernels of semi-separable class. In particular, under arbitrary well-posed boundary condi…
We study gradient-based optimization methods obtained by directly discretizing a second-order ordinary differential equation (ODE) related to the continuous limit of Nesterov's accelerated gradient method. When the function is smooth enough, we show that acceleration can be achieved by a stable discretization of this ODE using standard Runge-Kutta integrators. Specifically, we prove that under Lipschitz-gradient, convexity and order-$(s+2)$ differentiability assumptions, the sequence of iterates generated by discretizing the proposed second-order ODE converges to the optimal solution at a rate…
Although Nesterov's accelerated gradient (NAG) methods have been studied from various perspectives, it remains unclear why the most popular forms of NAG must handle convex and strongly convex objective functions separately. Motivated by this inconsistency, we propose an NAG method that unifies the existing ones for the convex and strongly convex cases. We first design a Lagrangian function that continuously extends the first Bregman Lagrangian to the strongly convex setting. As a specific case of the Euler--Lagrange equation for this Lagrangian, we derive an ordinary differential equation (ODE…
A numerical study of an algorithm proposed by Gusein Guseinov, which determines approximations to the optimal solution of problems of calculus of variations using two discretizations and correspondent Euler-Lagrange equations, is investigated. The results we obtain to discretizations of the brachistochrone problem and Mania example with Lavrentiev's phenomenon are compared with the solutions found by other methods and solvers. We conclude that Guseinov's method presents better solutions in most of the cases studied.
Layerwise goal-oriented adaptivity for neural ODEs: an optimal control perspective
Institute for Mathematics, Humboldt-Universität zu Berlin, Berlin, Germany（柏林洪堡大学数学研究所）；Weierstrass Institute for Applied Analysis and Stochastics, Berlin, Germany（魏泽拉施应用分析与随机学研究所）；Mathematical Institute, Universität Koblenz, Koblenz, Germany（科布伦茨大学数学研究所）
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of California, Santa Barbara（加利福尼亚大学圣巴巴拉分校）
Much recent interest has focused on the design of optimization algorithms from the discretization of an associated optimization flow, i.e., a system of differential equations (ODEs) whose trajectories solve an associated optimization problem. Such a design approach poses an important problem: how to find a principled methodology to design and discretize appropriate ODEs. This paper aims to provide a solution to this problem through the use of contraction theory. We first introduce general mathematical results that explain how contraction theory guarantees the stability of the implicit and expl…
A recent paradigm views deep neural networks as discretizations of certain controlled ordinary differential equations, sometimes called neural ordinary differential equations. We make use of this perspective to link expressiveness of deep networks to the notion of controllability of dynamical systems. Using this connection, we study an expressiveness property that we call universal interpolation, and show that it is generic in a certain sense. The universal interpolation property is slightly weaker than universal approximation, and disentangles supervised learning on finite training sets from …
In this work we propose a differential geometric motivation for Nesterov's accelerated gradient method (AGM) for strongly-convex problems. By considering the optimization procedure as occurring on a Riemannian manifold with a natural structure, The AGM method can be seen as the proximal point method applied in this curved space. This viewpoint can also be extended to the continuous time case, where the accelerated gradient method arises from the natural block-implicit Euler discretization of an ODE on the manifold. We provide an analysis of the convergence rate of this ODE for quadratic object…
