Gauss-Newton Temporal Difference Learning with Nonlinear Function Approximation
Center for Data Science, Peking University（北京大学数据科学中心）；National University of Singapore（新加坡国立大学）；Beijing International Center for Mathematical Research, Peking University（北京国际数学研究中心（北京大学））
In this paper, a Gauss-Newton Temporal Difference (GNTD) learning method is proposed to solve the Q-learning problem with nonlinear function approximation. In each iteration, our method takes one Gauss-Newton (GN) step to optimize a variant of Mean-Squared Bellman Error (MSBE), where target networks are adopted to avoid double sampling. Inexact GN steps are analyzed so that one can safely and efficiently compute the GN updates by cheap matrix iterations. Under mild conditions, non-asymptotic finite-sample convergence to the globally optimal Q function is derived for various nonlinear function approximations. In particular, for neural network parameterization with relu activation, GNTD achieves an improved sample complexity of $\tilde{\mathcal{O}}(\varepsilon^{-1})$, as opposed to the $\mathcal{\mathcal{O}}(\varepsilon^{-2})$ sample complexity of the existing neural TD methods. An $\tilde{\mathcal{O}}(\varepsilon^{-1.5})$ sample complexity of GNTD is also established for general smooth function approximations. We validate our method via extensive experiments in several RL benchmarks, where GNTD exhibits both higher rewards and faster convergence than TD-type methods.
Approximate Newton methods are a standard optimization tool which aim to maintain the benefits of Newton's method, such as a fast rate of convergence, whilst alleviating its drawbacks, such as computationally expensive calculation or estimation of the inverse Hessian. In this work we investigate approximate Newton methods for policy optimization in Markov Decision Processes (MDPs). We first analyse the structure of the Hessian of the objective function for MDPs. We show that, like the gradient, the Hessian exhibits useful structure in the context of MDPs and we use this analysis to motivate tw…
Computationally efficient Gauss-Newton reinforcement learning for model predictive control
Chair of Process Automation Systems, TU Dortmund University（多特蒙德工业大学过程自动化系统教席）；Department of Engineering Cybernetics, Norwegian University of Science and Technology (NTNU)（挪威科技大学工程控制论系）
Gauss-Newton Natural Gradient Descent for Shape Learning
Seminar for Applied Mathematics (SAM), D-Math, ETH Zürich（应用数学研讨会（SAM）、D-Math、苏黎世联邦理工学院）；Institute for Machine Learning, JKU Linz（机器学习研究所、杰克逊维尔大学）；ETH AI Center, ETH Zürich（苏黎世联邦理工学院人工智能中心）
Under the hypothesis that an initial point is a quasi-regular point, we use a majorant condition to present a new semi-local convergence analysis of an extension of the Gauss-Newton method for solving convex composite optimization problems. In this analysis the conditions and proof of convergence are simplified by using a simple majorant condition to define regions where a Gauss-Newton sequence is "well behaved".
Incremental Gauss--Newton Methods with Superlinear Convergence Rates
School of Data Science, Fudan University（复旦大学数据科学学院）；National University of Singapore（新加坡国立大学）；Department of Computer Science and Engineering, The Chinese University of Hong Kong（香港中文大学计算机科学与工程学系）
This work presents a novel version of recently developed Gauss-Newton method for solving systems of nonlinear equations, based on upper bound of solution residual and quadratic regularization ideas. We obtained for such method global convergence bounds and under natural non-degeneracy assumptions we present local quadratic convergence results. We developed stochastic optimization algorithms for presented Gauss-Newton method and justified sub-linear and linear convergence rates for these algorithms using weak growth condition (WGC) and Polyak-Lojasiewicz (PL) inequality. We show that Gauss-Newt…
In this paper, we study the Gauss-Newton method for a special class of systems of nonlinear equation. Under the hypothesis that the derivative of the function under consideration satisfies a majorant condition, semi-local convergence analysis is presented. In this analysis the conditions and proof of convergence are simplified by using a simple majorant condition to define regions where the Gauss-Newton sequence is "well behaved". Moreover, special cases of the general theory are presented as applications.
In this paper, inexact Gauss-Newton like methods for solving injective-overdetermined systems of equations are studied. We use a majorant condition, defined by a function whose derivative is not necessarily convex, to extend and improve several existing results on the local convergence of the Gauss-Newton methods.In particular, this analysis guarantees the convergence of the methods for two important new cases.
University of Oxford（牛津大学）；The Alan Turing Institute for Data Science（艾伦·图灵数据科学研究所）
We propose a Randomised Subspace Gauss-Newton (R-SGN) algorithm for solving nonlinear least-squares optimization problems, that uses a sketched Jacobian of the residual in the variable domain and solves a reduced linear least-squares on each iteration. A sublinear global rate of convergence result is presented for a trust-region variant of R-SGN, with high probability, which matches deterministic counterpart results in the order of the accuracy tolerance. Promising preliminary numerical results are presented for R-SGN on logistic regression and on nonlinear regression problems from the CUTEst …
In this paper, we present a local convergence analysis of inexact Gauss-Newton like methods for solving nonlinear least squares problems. Under the hypothesis that the derivative of the function associated with the least square problem satisfies a majorant condition, we obtain that the method is well-defined and converges. Our analysis provides a clear relationship between the majorant function and the function associated with the least square problem. It also allows us to obtain an estimate of convergence ball for inexact Gauss-Newton like methods and some important, special cases.
The Gauss-Newton's method for solving nonlinear least squares problems is studied in this paper. Under the hypothesis that the derivative of the function associated with the least square problem satisfies a majorant condition, a local convergence analysis is presented. This analysis allow us to obtain the optimal convergence radius, the biggest range for the uniqueness of solution, and to unify two previous and unrelated results.
This article studies Gauss-Newton-type methods for over-determined systems to find solutions to bilevel programming problems. To proceed, we use the lower-level value function reformulation of bilevel programs and consider necessary optimality conditions under appropriate assumptions. First under strict complementarity for upper- and lower-level feasibility constraints, we prove the convergence of a Gauss-Newton-type method in computing points satisfying these optimality conditions under additional tractable qualification conditions. Potential approaches to address the shortcomings of the meth…
We present examples of divergence for the BFGS and Gauss Newton methods. These examples have objective functions with bounded level sets and other properties concerning the examples published recently in this journal, like unit steps and convexity along the search lines. As these other examples, the iterates, function values and gradients in the new examples fit into the general formulation in our previous work {\it On the divergence of line search methods, Comput. Appl. Math. vol.26 no.1 (2007)}, which also presents an example of divergence for Newton's method.
In this paper, the proximal Gauss-Newton method for solving penalized nonlinear least squares problems is studied. A local convergence analysis is obtained under the assumption that the derivative of the function associated with the penalized least square problem satisfies a majorant condition. Our analysis provides a clear relationship between the majorant function and the function associated with the penalized least squares problem. The convergence for two important special cases is also derived.
A local convergence analysis of the Gauss-Newton method for solving injective-overdetermined systems of nonlinear equations under a majorant condition is provided. The convergence as well as results on its rate are established without a convexity hypothesis on the derivative of the majorant function. The optimal convergence radius, the biggest range for uniqueness of the solution along with some other special cases are also obtained.range for uniqueness of the solution along with some other special cases.
We present a novel distributed Gauss-Newton method for the non-linear state estimation (SE) model based on a probabilistic inference method called belief propagation (BP). The main novelty of our work comes from applying BP sequentially over a sequence of linear approximations of the SE model, akin to what is done by the Gauss-Newton method. The resulting iterative Gauss-Newton belief propagation (GN-BP) algorithm can be interpreted as a distributed Gauss-Newton method with the same accuracy as the centralized SE, however, introducing a number of advantages of the BP framework. The paper provi…
We analyze a preconditioned subgradient method for optimizing composite functions $h \circ c$, where $h$ is a locally Lipschitz function and $c$ is a smooth nonlinear mapping. We prove that when $c$ satisfies a constant rank property and $h$ is semismooth and sharp on the image of $c$, the method converges linearly. In contrast to standard subgradient methods, its oracle complexity is invariant under reparameterizations of $c$.
In this paper, we develop a variant of the well-known Gauss-Newton (GN) method to solve a class of nonconvex optimization problems involving low-rank matrix variables. As opposed to the standard GN method, our algorithm allows one to handle general smooth convex objective function. We show, under mild conditions, that the proposed algorithm globally and locally converges to a stationary point of the original problem. We also show empirically that the GN algorithm achieves higher accurate solutions than the alternating minimization algorithm (AMA). Then, we specify our GN scheme to handle the s…
We present a sparse Gauss-Newton solver for accelerated sensitivity analysis with applications to a wide range of equilibrium-constrained optimization problems. Dense Gauss-Newton solvers have shown promising convergence rates for inverse problems, but the cost of assembling and factorizing the associated matrices has so far been a major stumbling block. In this work, we show how the dense Gauss-Newton Hessian can be transformed into an equivalent sparse matrix that can be assembled and factorized much more efficiently. This leads to drastically reduced computation times for many inverse probl…
We propose a globally convergent Gauss-Newton algorithm for finding a local optimal solution of a non-convex and possibly non-smooth optimization problem. The algorithm that we present is based on a Gauss-Newton-type iteration for the non-smooth penalized formulation of the original problem. We establish a global convergence rate for this scheme from any initial point to a stationary point of the problem while using an exact penalty formulation. Under some more restrictive conditions we also derive local quadratic convergence for this scheme. We apply our proposed algorithm to solve the Altern…
As second-order methods, Gauss--Newton-type methods can be more effective than first-order methods for the solution of nonsmooth optimization problems with expensive-to-evaluate smooth components. Such methods, however, often do not converge. Motivated by nonlinear inverse problems with nonsmooth regularization, we propose a new Gauss--Newton-type method with inexact relaxed steps. We prove that the method converges to a set of disjoint critical points given that the linearisation of the forward operator for the inverse problem is sufficiently precise. We extensively evaluate the performance o…
Gaussian Mixture Models are a powerful tool in Data Science and Statistics that are mainly used for clustering and density approximation. The task of estimating the model parameters is in practice often solved by the Expectation Maximization (EM) algorithm which has its benefits in its simplicity and low per-iteration costs. However, the EM converges slowly if there is a large share of hidden information or overlapping clusters. Recent advances in manifold optimization for Gaussian Mixture Models have gained increasing interest. We introduce an explicit formula for the Riemannian Hessian for G…
In this paper, we propose a stochastic Gauss-Newton (SGN) algorithm to study the online principal component analysis (OPCA) problem, which is formulated by using the symmetric low-rank product (SLRP) model for dominant eigenspace calculation. Compared with existing OPCA solvers, SGN is of improved robustness with respect to the varying input data and algorithm parameters. In addition, turning to an evaluation of data stream based on approximated objective functions, we develop a new adaptive stepsize strategy for SGN (AdaSGN) which requires no priori knowledge of the input data, and numericall…
We present DFO-GN, a derivative-free version of the Gauss-Newton method for solving nonlinear least-squares problems. As is common in derivative-free optimization, DFO-GN uses interpolation of function values to build a model of the objective, which is then used within a trust-region framework to give a globally-convergent algorithm requiring $O(ε^{-2})$ iterations to reach approximate first-order criticality within tolerance $ε$. This algorithm is a simplification of the method from [H. Zhang, A. R. Conn, and K. Scheinberg, A Derivative-Free Algorithm for Least-Squares Minimization, SIAM J. O…
We develop two new stochastic Gauss-Newton algorithms for solving a class of non-convex stochastic compositional optimization problems frequently arising in practice. We consider both the expectation and finite-sum settings under standard assumptions, and use both classical stochastic and SARAH estimators for approximating function values and Jacobians. In the expectation case, we establish $\mathcal{O}(\varepsilon^{-2})$ iteration-complexity to achieve a stationary point in expectation and estimate the total number of stochastic oracle calls for both function value and its Jacobian, where $\v…
A q-Gauss-Newton algorithm is an iterative procedure that solves nonlinear unconstrained optimization problems based on minimization of the sum squared errors of the objective function residuals. Main advantage of the algorithm is that it approximates matrix of q-second order derivatives with the first-order q-Jacobian matrix. For that reason, the algorithm is much faster than q-steepest descent algorithms. The convergence of q-GN method is assured only when the initial guess is close enough to the solution. In this paper the influence of the parameter q to the non-linear problem solving is pr…
In this paper, we propose a solution to an AC state estimation problem in electric power systems using a fully distributed Gauss-Newton method. The proposed method is placed within the context of factor graphs and belief propagation algorithms and closed-form expressions for belief propagation messages exchanged along the factor graph are derived. The obtained algorithm provides the same solution as the conventional weighted least-squares state estimation. Using a simple example, we provide a step-by-step presentation of the proposed algorithm. Finally, we discuss the convergence behaviour usi…
First-order methods such as stochastic gradient descent (SGD) are currently the standard algorithm for training deep neural networks. Second-order methods, despite their better convergence rate, are rarely used in practice due to the prohibitive computational cost in calculating the second-order information. In this paper, we propose a novel Gram-Gauss-Newton (GGN) algorithm to train deep neural networks for regression problems with square loss. Our method draws inspiration from the connection between neural network optimization and kernel regression of neural tangent kernel (NTK). Different f…
In this paper, we study the placement of Phasor Measurement Units (PMU) for enhancing hybrid state estimation via the traditional Gauss-Newton method, which uses measurements from both PMU devices and Supervisory Control and Data Acquisition (SCADA) systems. To compare the impact of PMU placements, we introduce a useful metric which accounts for three important requirements in power system state estimation: {\it convergence}, {\it observability} and {\it performance} (COP). Our COP metric can be used to evaluate the estimation performance and numerical stability of the state estimator, which i…
In this paper, we investigate the convergence performance of a cooperative diffusion Gauss-Newton (GN) method, which is widely used to solve the nonlinear least squares problems (NLLS) due to the low computation cost compared with Newton's method. This diffusion GN collects the diversity of temporalspatial information over the network, which is used on local updates. In order to address the challenges on convergence analysis, we firstly consider to form a global recursion relation over spatial and temporal scales since the traditional GN is a time iterative method and the network-wide NLLS nee…
In this paper is proposed a novel incremental iterative Gauss-Newton-Markov-Kalman filter method for state estimation of dynamic models given noisy measurements. The mathematical formulation of the proposed filter is based on the construction of an optimal nonlinear map between the observable and parameter (state) spaces via a convergent sequence of linear maps obtained by successive linearisation of the observation operator in a Gauss-Newton-like form. To allow automatic linearisation of the dynamical system in a sparse form, the smoother is designed in a hierarchical setting such that the fo…
