We revisit three classical numerical methods for solving unconstrained optimal control problems - multiple shooting, single shooting, and differential dynamic programming - and examine their local convergence behaviour. In particular, we show that all three methods converge with the same linear rate if a Gauss-Newton (GN), or more general a Generalized Gauss-Newton (GGN), Hessian approximation is used, which is the case in widely used implementations such as iLQR.
In this article we propose a shooting algorithm for partially-affine optimal control problems, this is, systems in which the controls appear both linearly and nonlinearly in the dynamics. Since the shooting system generally has more equations than unknowns, the algorithm relies on the Gauss-Newton method. As a consequence, the convergence is locally quadratic provided that the derivative of the shooting function is injective and Lipschitz continuous at the optimal solution. We provide a proof of the convergence for the proposed algorithm using recently developed second order sufficient conditi…
This work presents a novel version of recently developed Gauss-Newton method for solving systems of nonlinear equations, based on upper bound of solution residual and quadratic regularization ideas. We obtained for such method global convergence bounds and under natural non-degeneracy assumptions we present local quadratic convergence results. We developed stochastic optimization algorithms for presented Gauss-Newton method and justified sub-linear and linear convergence rates for these algorithms using weak growth condition (WGC) and Polyak-Lojasiewicz (PL) inequality. We show that Gauss-Newt…
This article studies Gauss-Newton-type methods for over-determined systems to find solutions to bilevel programming problems. To proceed, we use the lower-level value function reformulation of bilevel programs and consider necessary optimality conditions under appropriate assumptions. First under strict complementarity for upper- and lower-level feasibility constraints, we prove the convergence of a Gauss-Newton-type method in computing points satisfying these optimality conditions under additional tractable qualification conditions. Potential approaches to address the shortcomings of the meth…
The divergence of the BFGS and Gauss Newton Methods
我们提出的例子发散的BFGS和高斯牛顿方法。这些例子的目标函数具有有界水平集和其他属性，这些属性与最近在这本杂志上发表的例子有关，如沿着搜索线的单位步长和凸性。作为这些其他示例，新示例中的迭代、函数值和梯度适合于我们先前工作{ \it On the divergence of line搜索法，Comput. Appl.Math.vol.26no.1（2007）}，其也给出了牛顿法的发散的例子。
We analyze a preconditioned subgradient method for optimizing composite functions $h \circ c$, where $h$ is a locally Lipschitz function and $c$ is a smooth nonlinear mapping. We prove that when $c$ satisfies a constant rank property and $h$ is semismooth and sharp on the image of $c$, the method converges linearly. In contrast to standard subgradient methods, its oracle complexity is invariant under reparameterizations of $c$.
We propose a globally convergent Gauss-Newton algorithm for finding a local optimal solution of a non-convex and possibly non-smooth optimization problem. The algorithm that we present is based on a Gauss-Newton-type iteration for the non-smooth penalized formulation of the original problem. We establish a global convergence rate for this scheme from any initial point to a stationary point of the problem while using an exact penalty formulation. Under some more restrictive conditions we also derive local quadratic convergence for this scheme. We apply our proposed algorithm to solve the Altern…
In this paper, we propose a stochastic Gauss-Newton (SGN) algorithm to study the online principal component analysis (OPCA) problem, which is formulated by using the symmetric low-rank product (SLRP) model for dominant eigenspace calculation. Compared with existing OPCA solvers, SGN is of improved robustness with respect to the varying input data and algorithm parameters. In addition, turning to an evaluation of data stream based on approximated objective functions, we develop a new adaptive stepsize strategy for SGN (AdaSGN) which requires no priori knowledge of the input data, and numericall…
我们提出了DFO-GN，一个导数免费版本的高斯-牛顿法求解非线性最小二乘问题。与无导数优化中常见的一样，DFO-GN使用函数值的插值来构建目标的模型，然后在信赖域框架内使用该模型来给出需要$O(ε^{-2})$迭代的全局收敛算法，以在公差$ε$内达到近似一阶临界性。该算法是[H. Zhang，中国古柏A. R. Conn和K.张文，最小二乘最小化算法的一种新算法，北京：计算机科学出版社，20（2010），pp. 3555-3576]，其中我们用线性模型替换每个残差的二次模型。我们证明了DFO-GN在客观评估方面与Zhang等人的方法相比，具有更快的运行时间和更高的可扩展性。
We propose an efficient way of solving optimal control problems for rigid-body systems on the basis of inverse dynamics and the multiple-shooting method. We treat all variables, including the state, acceleration, and control input torques, as optimization variables and treat the inverse dynamics as an equality constraint. We eliminate the update of the control input torques from the linear equation of Newton's method by applying condensing for inverse dynamics. The size of the resultant linear equation is the same as that of the multiple-shooting method based on forward dynamics except for the…
A Semismooth Newton-Based Proximal Augmented Lagrangian Method for Joint Estimation of Multiple Gaussian Graphical Models with Clustered Structure
School of Mathematics and Statistics, Fuzhou University（福州大学数学与统计学院）；Center for Applied Mathematics of Fujian Province, School of Mathematics and Statistics, Fuzhou University（福建省应用数学中心，福州大学数学与统计学院）
In this paper is proposed a novel incremental iterative Gauss-Newton-Markov-Kalman filter method for state estimation of dynamic models given noisy measurements. The mathematical formulation of the proposed filter is based on the construction of an optimal nonlinear map between the observable and parameter (state) spaces via a convergent sequence of linear maps obtained by successive linearisation of the observation operator in a Gauss-Newton-like form. To allow automatic linearisation of the dynamical system in a sparse form, the smoother is designed in a hierarchical setting such that the fo…
In this paper, we develop a variant of the well-known Gauss-Newton (GN) method to solve a class of nonconvex optimization problems involving low-rank matrix variables. As opposed to the standard GN method, our algorithm allows one to handle general smooth convex objective function. We show, under mild conditions, that the proposed algorithm globally and locally converges to a stationary point of the original problem. We also show empirically that the GN algorithm achieves higher accurate solutions than the alternating minimization algorithm (AMA). Then, we specify our GN scheme to handle the s…
We present a novel distributed Gauss-Newton method for the non-linear state estimation (SE) model based on a probabilistic inference method called belief propagation (BP). The main novelty of our work comes from applying BP sequentially over a sequence of linear approximations of the SE model, akin to what is done by the Gauss-Newton method. The resulting iterative Gauss-Newton belief propagation (GN-BP) algorithm can be interpreted as a distributed Gauss-Newton method with the same accuracy as the centralized SE, however, introducing a number of advantages of the BP framework. The paper provi…
We present a sparse Gauss-Newton solver for accelerated sensitivity analysis with applications to a wide range of equilibrium-constrained optimization problems. Dense Gauss-Newton solvers have shown promising convergence rates for inverse problems, but the cost of assembling and factorizing the associated matrices has so far been a major stumbling block. In this work, we show how the dense Gauss-Newton Hessian can be transformed into an equivalent sparse matrix that can be assembled and factorized much more efficiently. This leads to drastically reduced computation times for many inverse probl…
Gauss-Newton Natural Gradient Descent for Shape Learning
Seminar for Applied Mathematics (SAM), D-Math, ETH Zürich（应用数学研讨会（SAM）、D-Math、苏黎世联邦理工学院）；Institute for Machine Learning, JKU Linz（机器学习研究所、杰克逊维尔大学）；ETH AI Center, ETH Zürich（苏黎世联邦理工学院人工智能中心）
We develop two new stochastic Gauss-Newton algorithms for solving a class of non-convex stochastic compositional optimization problems frequently arising in practice. We consider both the expectation and finite-sum settings under standard assumptions, and use both classical stochastic and SARAH estimators for approximating function values and Jacobians. In the expectation case, we establish $\mathcal{O}(\varepsilon^{-2})$ iteration-complexity to achieve a stationary point in expectation and estimate the total number of stochastic oracle calls for both function value and its Jacobian, where $\v…
A q-Gauss-Newton algorithm is an iterative procedure that solves nonlinear unconstrained optimization problems based on minimization of the sum squared errors of the objective function residuals. Main advantage of the algorithm is that it approximates matrix of q-second order derivatives with the first-order q-Jacobian matrix. For that reason, the algorithm is much faster than q-steepest descent algorithms. The convergence of q-GN method is assured only when the initial guess is close enough to the solution. In this paper the influence of the parameter q to the non-linear problem solving is pr…
As second-order methods, Gauss--Newton-type methods can be more effective than first-order methods for the solution of nonsmooth optimization problems with expensive-to-evaluate smooth components. Such methods, however, often do not converge. Motivated by nonlinear inverse problems with nonsmooth regularization, we propose a new Gauss--Newton-type method with inexact relaxed steps. We prove that the method converges to a set of disjoint critical points given that the linearisation of the forward operator for the inverse problem is sufficiently precise. We extensively evaluate the performance o…
Incremental Gauss--Newton Methods with Superlinear Convergence Rates
School of Data Science, Fudan University（复旦大学数据科学学院）；National University of Singapore（新加坡国立大学）；Department of Computer Science and Engineering, The Chinese University of Hong Kong（香港中文大学计算机科学与工程学系）
University of Oxford（牛津大学）；The Alan Turing Institute for Data Science（艾伦·图灵数据科学研究所）
We propose a Randomised Subspace Gauss-Newton (R-SGN) algorithm for solving nonlinear least-squares optimization problems, that uses a sketched Jacobian of the residual in the variable domain and solves a reduced linear least-squares on each iteration. A sublinear global rate of convergence result is presented for a trust-region variant of R-SGN, with high probability, which matches deterministic counterpart results in the order of the accuracy tolerance. Promising preliminary numerical results are presented for R-SGN on logistic regression and on nonlinear regression problems from the CUTEst …
In this paper, we investigate the convergence performance of a cooperative diffusion Gauss-Newton (GN) method, which is widely used to solve the nonlinear least squares problems (NLLS) due to the low computation cost compared with Newton's method. This diffusion GN collects the diversity of temporalspatial information over the network, which is used on local updates. In order to address the challenges on convergence analysis, we firstly consider to form a global recursion relation over spatial and temporal scales since the traditional GN is a time iterative method and the network-wide NLLS nee…
Gauss-Newton Temporal Difference Learning with Nonlinear Function Approximation
Center for Data Science, Peking University（北京大学数据科学中心）；National University of Singapore（新加坡国立大学）；Beijing International Center for Mathematical Research, Peking University（北京国际数学研究中心（北京大学））
First-order methods such as stochastic gradient descent (SGD) are currently the standard algorithm for training deep neural networks. Second-order methods, despite their better convergence rate, are rarely used in practice due to the prohibitive computational cost in calculating the second-order information. In this paper, we propose a novel Gram-Gauss-Newton (GGN) algorithm to train deep neural networks for regression problems with square loss. Our method draws inspiration from the connection between neural network optimization and kernel regression of neural tangent kernel (NTK). Different f…
