We propose a continuous-time second-order optimization algorithm for solving unconstrained convex optimization problems with bounded Hessian. We show that this alternative algorithm has a comparable convergence rate to that of the continuous-time Newton-Raphson method, however structurally, it is amenable to a more efficient distributed implementation. We present a distributed implementation of our proposed optimization algorithm and prove its convergence via Lyapunov analysis. A numerical example demonstrates our results.
We extend a multi-agent convex-optimization algorithm named Newton-Raphson consensus to a network scenario that involves directed, asynchronous and lossy communications. We theoretically analyze the stability and performance of the algorithm and, in particular, provide sufficient conditions that guarantee local exponential convergence of the node-states to the global centralized minimizer even in presence of packet losses. Finally, we complement the theoretical analysis with numerical simulations that compare the performance of the Newton-Raphson consensus against asynchronous implementations …
In this work we study the problem of unconstrained convex-optimization in a fully distributed multi-agent setting which includes asynchronous computation and lossy communication. In particular, we extend a recently proposed algorithm named Newton-Raphson Consensus by integrating it with a broadcast-based average consensus algorithm which is robust to packet losses. We show via the separation of time scales principle that under mild conditions (i.e., persistency of the agents activation and bounded consecutive communication failures) the proposed algorithm is proved to be locally exponentially …
We address the problem of distributed uncon- strained convex optimization under separability assumptions, i.e., the framework where each agent of a network is endowed with a local private multidimensional convex cost, is subject to communication constraints, and wants to collaborate to compute the minimizer of the sum of the local costs. We propose a design methodology that combines average consensus algorithms and separation of time-scales ideas. This strategy is proved, under suitable hypotheses, to be globally convergent to the true minimizer. Intuitively, the procedure lets the agents dist…
A Hessian based optimal control method is presented in Liouville space to mitigate previously undesirable polynomial scaling of computation time. This new method, an improvement to the state-of-the-art Newton-Raphson GRAPE method, is derived with respect to two exact time-propagator derivative techniques: auxiliary matrix and ESCALADE methods. We observed that compared to the best current implementation of Newton-Raphson GRAPE method, for an ensemble of 2-level systems, with realistic conditions, the new auxiliary matrix and ESCALADE Hessians can be 4-200 and 70-600 times faster, respectively.
This paper presents a control technique for output tracking of reference signals in continuous-time dynamical systems. The technique is comprised of the following three elements: (i) output prediction which has to track the reference signal, (ii) a controller based on an integrator with variable gain, and (iii) a speedup of the control action for enhancing the tracker's accuracy and, in some cases, guaranteeing stability of the closed-loop system. The technique is suitable for linear and nonlinear systems, implementable by simple algorithms, can track reference points as well as time-dependent…
This paper focuses on the AC Optimal Power Flow (OPF) problem for multi-phase systems. Particular emphasis is given to systems with high integration of renewables, where adjustments of the real and reactive output powers from renewable sources of energy are necessary in order to enforce voltage regulation. The AC OPF problem is known to be nonconvex (and, in fact, NP-hard). Convex relaxation techniques have been recently explored to solve the OPF task with reduced computational burden; however, sufficient conditions for tightness of these relaxations are only available for restricted classes o…
We propose a new globally convergent stochastic second order method. Our starting point is the development of a new Sketched Newton-Raphson (SNR) method for solving large scale nonlinear equations of the form $F(x)=0$ with $F:\mathbb{R}^p \rightarrow \mathbb{R}^m$. We then show how to design several stochastic second order optimization methods by re-writing the optimization problem of interest as a system of nonlinear equations and applying SNR. For instance, by applying SNR to find a stationary point of a generalized linear model (GLM), we derive completely new and scalable stochastic second …
Newton-Raphson Flow for Aggressive Quadrotor Tracking Control
Georgia Institute of Technology（佐治亚理工学院）；School of Electrical and Computer Engineering（电气与计算机工程学院）；School of Civil and Environmental Engineering（土木与环境工程学院）
The identification of pipe roughnesses in a water distribution network is formulated as nonlinear system of algebraic equations which turns out to be demanding to solve under real-world circumstances. This paper proposes an enhanced technique to numerically solve this identification problem, extending the conventional Newton-Raphson approach with second-order derivatives in the determination of the search direction. Enabled through some interesting mathematical findings, the resulting formulation can be represented compactly and thus facilitates the development of an efficient and more robust …
The paper aims to find the solution of oligopoly market equilibrium problem through system of nonlinear equations. We propose modified newton method to obtain the solution of system of nonlinear equations. We show that our proposed method has higher order of convergence. Keywords: System of nonlinear equations, modified newton method, oligopolistic market equilibrium problem.
In this paper, we formulate the Load Flow (LF) problem in radial electricity distribution networks as an unconstrained Riemannian optimization problem, consisting of two manifolds, and we consider alternative retractions and initialization options. Our contribution is a novel LF solution method, which we show belongs to the family of Riemannian approximate Newton methods guaranteeing monotonic descent and local superlinear convergence rate. To the best of our knowledge, this is the first exact LF solution method employing Riemannian optimization. Extensive numerical comparisons on several test…
We consider the efficient minimization of a nonlinear, strictly convex functional with $\ell_1$-penalty term. Such minimization problems appear in a wide range of applications like Tikhonov regularization of (non)linear inverse problems with sparsity constraints. In (2015 Inverse Problems (31) 025005), a globalized Bouligand-semismooth Newton method was presented for $\ell_1$-Tikhonov regularization of linear inverse problems. Nevertheless, a technical assumption on the accumulation point of the sequence of iterates was necessary to prove global convergence. Here, we generalize this method to …
Redwood Center for Theoretical Neuroscience（雷德伍德理论神经科学中心）；University of California, Berkeley（加州大学伯克利分校）
Understanding of the behavior of algorithms for resolving the optimization problem (hereafter shortened to OP) of optimizing a differentiable loss function (OP1), is enhanced by knowledge of the critical points of that loss function, i.e. the points where the gradient is 0. Here, we describe a solution to the problem of finding critical points by proposing and solving three optimization problems: 1) minimizing the norm of the gradient (OP2), 2) minimizing the difference between the pre-conditioned update direction and the gradient (OP3), and 3) minimizing the norm of the gradient along the upd…
Shizuoka University（静冈大学）；Shonan Institute of Technology（湘南工科大学）；Hiroshima City University（广岛市立大学）
The Nesterov's accelerated quasi-Newton (L)NAQ method has shown to accelerate the conventional (L)BFGS quasi-Newton method using the Nesterov's accelerated gradient in several neural network (NN) applications. However, the calculation of two gradients per iteration increases the computational cost. The Momentum accelerated Quasi-Newton (MoQ) method showed that the Nesterov's accelerated gradient can be approximated as a linear combination of past gradients. This abstract extends the MoQ approximation to limited memory NAQ and evaluates the performance on a function approximation problem.
