H. Chen, H. C. Wu, S. C. Chan, W. H. Lam
This paper proposes a novel stochastic version of damped and regularized BFGS method for addressing the above problems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
继续探索
We study stochastic inexact Newton methods and consider their application in nonconvex settings. Building on the work of [R. Bollapragada, R. H. Byrd, and J. Nocedal, IMA Journal of Numerical Analysis, 39 (2018), pp. 545--578] we derive bounds for convergence rates in expected value for stochastic low rank Newton methods, and stochastic inexact Newton Krylov methods. These bounds quantify the errors incurred in subsampling the Hessian and gradient, as well as in approximating the Newton linear solve, and in choosing regularization and step length parameters. We deploy these methods in training…
Complexity Guarantees for Nonconvex Newton-MR Under Inexact Hessian Information
该研究扩展了非凸无约束优化的Newton-MR算法至非精确Hessian场景，分析了不同非凸条件下的复杂度与收敛性，并通过机器学习问题实验验证了算法的鲁棒性。
New Globalized Newton-Type Methods for Nonconvex Optimization Problems
针对非凸优化问题，提出通用线搜索牛顿框架，避免重复海森正则化，在温和假设下建立全局收敛性，适当条件下证明局部收敛性，应用于强拟凸优化并给出算法及分析，数值实验验证方法有效性。
This paper deals with regularized Newton methods, a flexible class of unconstrained optimization algorithms that is competitive with line search and trust region methods and potentially combines attractive elements of both. The particular focus is on combining regularization with limited memory quasi-Newton methods by exploiting the special structure of limited memory algorithms. Global convergence of regularization methods is shown under mild assumptions and the details of regularized limited memory quasi-Newton updates are discussed including their compact representations. Numerical results …
In this paper we consider finding an approximate second-order stationary point (SOSP) of nonconvex conic optimization that minimizes a twice differentiable function over the intersection of an affine subspace and a convex cone. In particular, we propose a Newton-conjugate gradient (Newton-CG) based barrier method for finding an $(ε,\sqrtε)$-SOSP of this problem. Our method is not only implementable, but also achieves an iteration complexity of ${\cal O}(ε^{-3/2})$, which matches the best known iteration complexity of second-order methods for finding an $(ε,\sqrtε)$-SOSP of unconstrained noncon…
In this paper we consider finding a second-order stationary point (SOSP) of nonconvex equality constrained optimization when a nearly feasible point is known. In particular, we first propose a new Newton-CG method for finding an approximate SOSP of unconstrained optimization and show that it enjoys a substantially better complexity than the Newton-CG method [56]. We then propose a Newton-CG based augmented Lagrangian (AL) method for finding an approximate SOSP of nonconvex equality constrained optimization, in which the proposed Newton-CG method is used as a subproblem solver. We show that und…
In this paper, we consider high-dimensional nonconvex square-root-loss regression problems and introduce a proximal majorization-minimization (PMM) algorithm for these problems. Our key idea for making the proposed PMM to be efficient is to develop a sparse semismooth Newton method to solve the corresponding subproblems. By using the Kurdyka-Łojasiewicz property exhibited in the underlining problems, we prove that the PMM algorithm converges to a d-stationary point. We also analyze the oracle property of the initial subproblem used in our algorithm. Extensive numerical experiments are presente…
A Newton-CG based barrier-augmented Lagrangian method for general nonconvex conic optimization
本文针对一般非凸锥优化问题提出基于Newton-CG的障碍增广拉格朗日方法，首次给出求解近似二阶稳定点的复杂度界，并验证其解质量优于一阶方法。
Stochastic Quasi-Newton Methods for Nonconvex Stochastic Optimization
本文研究非凸随机优化中的随机拟牛顿方法，提出了一种框架并证明了收敛性，分析了最坏情况下的迭代复杂度，提出了一种随机阻尼L-BFGS方法，并结合SVRG技术，展示了在非凸二分类和多分类问题中的数值结果。
In modern deep learning, highly subsampled stochastic approximation (SA) methods are preferred to sample average approximation (SAA) methods because of large data sets as well as generalization properties. Additionally, due to perceived costs of forming and factorizing Hessians, second order methods are not used for these problems. In this work we motivate the extension of Newton methods to the SA regime, and argue for the use of the scalable low rank saddle free Newton (LRSFN) method, which avoids forming the Hessian in favor of making a low rank approximation. Additionally, LRSFN can facilit…
We present Optimization Engine (OpEn): an open-source code generation tool for real-time embedded nonconvex optimization, which implements a novel numerical method. OpEn combines the proximal averaged Newton-type method for optimal control (PANOC) with the penalty and augmented Lagrangian methods to compute approximate stationary points of nonconvex problems. The proposed method involves very simple algebraic operations such as vector products, has a low memory footprint and exhibits very good convergence properties that allow the solution of nonconvex problems on embedded devices. OpEn's core…
Regularized methods via cubic model subspace minimization for nonconvex optimization
针对非凸优化自适应三次正则化方法的试探步计算问题，提出低维子空间重复利用的三次模型最小化新方法，辅以正则化牛顿步，保留经典方法最坏情况复杂度，实验表明其相比标准实现优势显著。
A Conservation Law Method in Optimization
本文提出基于牛顿第二定律无摩擦的算法，用于寻找非凸优化的局部极小值和某些程度的全局极小值，通过速度可观测性和可控性实现高效收敛。
In large-scale optimization, when either forming or storing Hessian matrices are prohibitively expensive, quasi-Newton methods are often used in lieu of Newton's method because they only require first-order information to approximate the true Hessian. Multipoint symmetric secant (MSS) methods can be thought of as generalizations of quasi-Newton methods in that they attempt to impose additional requirements on their approximation of the Hessian. Given an initial Hessian approximation, MSS methods generate a sequence of possibly-indefinite matrices using rank-2 updates to solve nonconvex unconst…
Quasi-Newton techniques approximate the Newton step by estimating the Hessian using the so-called secant equations. Some of these methods compute the Hessian using several secant equations but produce non-symmetric updates. Other quasi-Newton schemes, such as BFGS, enforce symmetry but cannot satisfy more than one secant equation. We propose a new type of quasi-Newton symmetric update using several secant equations in a least-squares sense. Our approach generalizes and unifies the design of quasi-Newton updates and satisfies provable robustness guarantees.
In this paper, we analyze the convergence of the alternating direction method of multipliers (ADMM) for minimizing a nonconvex and possibly nonsmooth objective function, $ϕ(x_0,\ldots,x_p,y)$, subject to coupled linear equality constraints. Our ADMM updates each of the primal variables $x_0,\ldots,x_p,y$, followed by updating the dual variable. We separate the variable $y$ from $x_i$'s as it has a special role in our analysis. The developed convergence guarantee covers a variety of nonconvex functions such as piecewise linear functions, $\ell_q$ quasi-norm, Schatten-$q$ quasi-norm ($0<q<1$), m…
In this paper, we introduce a proximal-proximal majorization-minimization (PPMM) algorithm for nonconvex tuning-free robust regression problems. The basic idea is to apply the proximal majorization-minimization algorithm to solve the nonconvex problem with the inner subproblems solved by a sparse semismooth Newton (SSN) method based proximal point algorithm (PPA). We must emphasize that the main difficulty in the design of the algorithm lies in how to overcome the singular difficulty of the inner subproblem. Furthermore, we also prove that the PPMM algorithm converges to a d-stationary point. …
Superlinear convergence has been an elusive goal for black-box nonsmooth optimization. Even in the convex case, the subgradient method is very slow, and while some cutting plane algorithms, including traditional bundle methods, are popular in practice, local convergence is still sluggish. Faster variants depend either on problem structure or on analyses that elide sequences of "null" steps. Motivated by a semi-structured approach to optimization and the sequential quadratic programming philosophy, we describe a new bundle Newton method that incorporates second-order objective information with …
A Regularized Newton Method for Computing Ground States of Bose-Einstein condensates
本文提出一种正则牛顿方法用于计算玻色-爱因斯坦凝聚态基态，通过有限差分、正弦或傅里叶伪谱离散化将能量泛函和约束离散化，结合可行梯度法和多级网格技术，实现高效准确的求解。
A modified BFGS quasi-Newton iterative formula
本文利用非线性多项式方程与雅可比矩阵的关系式，提出精确替代拟牛顿方程的方法，并推导出修正的BFGS更新公式。
The cubic regularization method (CR) and its adaptive version (ARC) are popular Newton-type methods in solving unconstrained non-convex optimization problems, due to its global convergence to local minima under mild conditions. The main aim of this paper is to develop a momentum-accelerated adaptive cubic regularization method (ARCm) to improve the convergent performance. With the proper choice of momentum step size, we show the global convergence of ARCm and the local convergence can also be guaranteed under the \KL property. Such global and local convergence can also be established when inex…
On efficiently computing the eigenvalues of limited-memory quasi-Newton matrices
本文提出了一种高效计算有限记忆拟牛顿矩阵特征值的方法，通过改进的QR分解提升计算效率，并在数值实验中验证了其高精度性能，适用于大规模优化和矩阵条件数计算。
Disguised and new Quasi-Newton methods for nonlinear eigenvalue problems
本文提出了一种针对非线性特征值问题的准牛顿方法，分析了不同雅可比矩阵近似对算法性能的影响，并建立了收敛理论。
An $LDL^T$ Trust-Region Quasi-Newton Method
针对无约束极小化，本文提出一种更新 $LDL^T$ 分解的信赖域拟牛顿法，在保持鲁棒性的同时将计算复杂度控制为问题规模的二次增长，并达到与传统线搜索方法相当的性能。
Combining Nonlinear FETI-DP Methods and Quasi-Newton Methods using an SQP Approach
本文研究基于SQP方法结合非线性FETI-DP与拟牛顿方法求解非线性有限元问题，通过拟牛顿近似海森矩阵（初始用精确矩阵并重启加速收敛），并提供了非线性结构力学模型的数值实验验证。
Quasi-Newton Approach for an Atmospheric Tomography Problem
本文研究了在准牛顿方法中使用已知的平滑总变分正则化方法解决GPS反演问题，通过准牛顿方法求解与一般Tikhonov泛函相关的非约束凸光滑最小化问题，并发现有限记忆BFGS算法结合信任区域能有效获得合理最优解。
We describe stochastic Newton and stochastic quasi-Newton approaches to efficiently solve large linear least-squares problems where the very large data sets present a significant computational burden (e.g., the size may exceed computer memory or data are collected in real-time). In our proposed framework, stochasticity is introduced in two different frameworks as a means to overcome these computational limitations, and probability distributions that can exploit structure and/or sparsity are considered. Theoretical results on consistency of the approximations for both the stochastic Newton and …
The Newton-Shamanskii method for solving a quadratic matrix equation arising in quasi-birth-death problems
本文提出利用牛顿-沙马恩斯基方法求解二次矩阵方程，证明该方法生成的矩阵序列单调递增并收敛到最小非负解，通过数值实验验证了方法的有效性。
An Efficient Quasi-Newton Method with Tensor Product Implementation for Solving Quasi-Linear Elliptic Equations and Systems
针对拟线性椭圆方程与方程组求解效率问题，提出结合线性拉普拉斯与简化非线性项近似雅可比的拟牛顿法，经收敛分析与二维三维数值实验验证，可借助GPU张量积实现提升计算效率，支撑多领域复杂仿真。
In this work, we address the implementation and performance of inexact Newton-Krylov and quasi-Newton algorithms, more specifically the BFGS method, for the solution of the nonlinear elasticity equations, and compare them to a standard Newton-Krylov method. This is done through a systematic analysis of the performance of the solvers with respect to the problem size, the magnitude of the data and the number of processors in both almost incompressible and incompressible mechanics. We consider three test cases: Cook's membrane (static, almost incompressible), a twist test (static, incompressible)…
Quasi-Newton Methods: A New Direction
本文通过将拟牛顿方法解释为贝叶斯线性回归的近似，揭示了经典算法的缺陷，并提出了一种新的非参数拟牛顿方法，在相似计算成本下更高效地利用信息。
扫码添加微信好友，提出您的宝贵建议 👇
💡 备注请填写：网站反馈
发表机构由 AI 从论文文本抽取，如发现错误，欢迎反馈修正建议。