我们扩展了传统信任区域方法在无约束、可能非凸优化中的复杂性分析。尽管大多数复杂性分析假设模型Hessian是均匀有界的，但我们处理的是可能无界的模型Hessian。在实际实现中，尤其是基于拟牛顿更新的算法如PSB、BFGS和SR1，有界性并不保证。我们考察了两种Hessian增长的模式：一种由成功迭代次数的幂次限制，另一种由总迭代次数的幂次限制。这使我们能够正式并解决Powell[IMA J. Numer. Ana. 30(1):289-301,2010]的直觉，他研究了在我们假设的特殊情况下收敛性，但其证明中包含复杂性论证。具体来说，对于0≤p<1，我们建立了找到ε-驻点的尖锐复杂性O([(1-p)ε^{-2}]^{1/(1-p)})，当模型Hessian为O(|S_{k-1}|^p)，其中|S_{k-1}|是到第k-1次迭代为止接受步数的次数。对于p=1的情况，即Powell研究的情况，我们建立了尖锐的复杂性O(exp(c_1ε^{-2}))，其中c_1>0是一个常数。这比Powell[2010]怀疑的双指数界要好得多，但比文献中其他推测的界要差。我们还建立了当模型Hessian为O(k^p)，其中k是迭代计数器时的类似尖锐界，对于0≤p<1。当p=1时，复杂度界取决于族的参数，但减少到O((1 - log(ε))exp(c_2ε^{-2}))，其中c_2>0是一个常数，对于标准信任区域方法的特殊情况。作为特殊情况，我们推导了在相同增长假设下的强凸目标函数的新型复杂度界。
英文摘要：
We extend traditional complexity analyses of trust-region methods for unconstrained, possibly nonconvex, optimization. Whereas most complexity analyses assume uniform boundedness of the model Hessians, we work with potentially unbounded model Hessians. Boundedness is not guaranteed in practical implementations, in particular ones based on quasi-Newton updates such as PSB, BFGS and SR1. We examine two regimes of Hessian growth: one bounded by a power of the number of successful iterations, and one bounded by a power of the number of iterations. This allows us to formalize and address the intuition of Powell [IMA J. Numer. Ana. 30(1):289-301,2010], who studied convergence under a special case of our assumptions, but whose proof contained complexity arguments. Specifically, for \(0 \leq p < 1\), we establish sharp \(O([(1-p)ε^{-2}]^{1/(1-p)})\) evaluation complexity to find an \(ε\)-stationary point when model Hessians are \(O(|\mathcal{S}_{k-1}|^p)\), where \(|\mathcal{S}_{k-1}|\) is the number of iterations where the step was accepted, up to iteration \(k-1\). For \(p = 1\), which is the case studied by Powell, we establish a sharp \(O(\exp(c_1ε^{-2}))\) evaluation complexity for a certain constant \(c_1 > 0\). This is far better than the double exponential bound that \citet{powell-2010} suspected, and is far worse than other bounds surmised elsewhere in the literature. We establish similar sharp bounds when model Hessians are \(O(k^p)\), where \(k\) is the iteration counter, for \(0 \leq p < 1\). When \(p = 1\), the complexity bound depends on the parameters of the family, but reduces to \(O((1 - \log(ε))\exp(c_2ε^{-2}))\) for a certain constant \(c_2 > 0\) for the special case of the standard trust-region method. As special cases, we derive novel complexity bounds for (strongly) convex objectives under the same growth assumptions.
A stochastic second-order trust region method is proposed, which can be viewed as a second-order extension of the trust-region-ish (TRish) algorithm proposed by Curtis et al. (INFORMS J. Optim. 1(3) 200-220, 2019). In each iteration, a search direction is computed by (approximately) solving a trust region subproblem defined by stochastic gradient and Hessian estimates. The algorithm has convergence guarantees for stochastic minimization in the fully stochastic regime, meaning that guarantees hold when each stochastic gradient is required merely to be an unbiased estimate of the true gradient w…
We target the problem of finding a local minimum in non-convex finite-sum minimization. Towards this goal, we first prove that the trust region method with inexact gradient and Hessian estimation can achieve a convergence rate of order $\mathcal{O}(1/{k^{2/3}})$ as long as those differential estimations are sufficiently accurate. Combining such result with a novel Hessian estimator, we propose the sample-efficient stochastic trust region (STR) algorithm which finds an $(ε, \sqrtε)$-approximate local minimum within $\mathcal{O}({\sqrt{n}}/{ε^{1.5}})$ stochastic Hessian oracle queries. This impr…
MATRO: Metric-Aware Trust-Region Optimization with Fully Quadratic Models
Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Lawrence Berkeley National Laboratory（劳伦斯伯克利国家实验室）
Sufficient Conditions for Error Distance Reduction in the (\ell^2)-norm Trust Region between Minimizers of Local Nonconvex Multivariate Quadratic Approximates
Convex and nonconvex finite-sum minimization arises in many scientific computing and machine learning applications. Recently, first-order and second-order methods where objective functions, gradients and Hessians are approximated by randomly sampling components of the sum have received great attention. We propose a new trust-region method which employs suitable approximations of the objective function, gradient and Hessian built via random subsampling techniques. The choice of the sample size is deterministic and ruled by the inexact restoration approach. We discuss local and global properties…
Recently, there has been a surge of interest in designing variants of the classical Newton-CG in which the Hessian of a (strongly) convex function is replaced by suitable approximations. This is mainly motivated by large-scale finite-sum minimization problems that arise in many machine learning applications. Going beyond convexity, inexact Hessian information has also been recently considered in the context of algorithms such as trust-region or (adaptive) cubic regularization for general non-convex problems. Here, we do that for Newton-MR, which extends the application range of the classical N…
This paper studies stability and symmetry preserving $H^2$ optimal model reduction problems of linear systems which include linear gradient systems as a special case. The problem is formulated as a nonlinear optimization problem on the product manifold of the manifold of symmetric positive definite matrices and the Euclidean spaces. To solve the problem by using the trust-region method, the gradient and Hessian of the objective function are derived. Furthermore, it is shown that if we restrict our systems to gradient systems, the gradient and Hessian can be obtained more efficiently. More conc…
In this work, we consider methods for large-scale and nonconvex unconstrained optimization. We propose a new trust-region method whose subproblem is defined using a so-called "shape-changing" norm together with densely-initialized multipoint symmetric secant (MSS) matrices to approximate the Hessian. Shape-changing norms and dense initializations have been successfully used in the context of traditional quasi-Newton methods, but have yet to be explored in the case of MSS methods. Numerical results suggest that trust-region methods that use densely-initialized MSS matrices together with shape-c…
Stanford University（斯坦福大学）；University of Queensland（昆士兰大学）；International Computer Science Institute（国际计算机科学研究所）；University of California at Berkeley（加州大学伯克利分校）
We consider variants of trust-region and cubic regularization methods for non-convex optimization, in which the Hessian matrix is approximated. Under mild conditions on the inexact Hessian, and using approximate solution of the corresponding sub-problems, we provide iteration complexity to achieve $ ε$-approximate second-order optimality which have shown to be tight. Our Hessian approximation conditions constitute a major relaxation over the existing ones in the literature. Consequently, we are able to show that such mild conditions allow for the construction of the approximate Hessian through…
Adaptive cubic regularization (ARC) methods for unconstrained optimization compute steps from linear systems involving a shifted Hessian in the spirit of the Levenberg-Marquardt and trust-region methods. The standard approach consists in performing an iterative search for the shift akin to solving the secular equation in trust-region methods. Such search requires computing the Cholesky factorization of a tentative shifted Hessian at each iteration, which limits the size of problems that can be reasonably considered. We propose a scalable implementation of ARC named ARCqK in which we solve a se…
The University of Sydney（悉尼大学）；University of California, Davis（加州大学戴维斯分校）
Trust region and cubic regularization methods have demonstrated good performance in small scale non-convex optimization, showing the ability to escape from saddle points. Each iteration of these methods involves computation of gradient, Hessian and function value in order to obtain the search direction and adjust the radius or cubic regularization parameter. However, exactly computing those quantities are too expensive in large-scale problems such as training deep networks. In this paper, we study a family of stochastic trust region and cubic regularization methods when gradient, Hessian and f…
Quasi-Newton and Krylov Methods for the Solution of Nonconvex Trust-Region Subproblems
National Institute of Applied Sciences of Rennes（法国国立应用科学学院）；GERAD and Department of Mathematics and Industrial Engineering, École Polytechnique（GERAD与蒙特利尔高等商学院数学与工业工程系）；Department of Mathematics, Université de Sherbrooke（舍布鲁克大学数学系）
Interpolation-based trust-region methods are an important class of algorithms for Derivative-Free Optimization which rely on locally approximating an objective function by quadratic polynomial interpolation models, frequently built from less points than there are basis components. Often, in practical applications, the contribution of the problem variables to the objective function is such that many pairwise correlations between variables are negligible, implying, in the smooth case, a sparse structure in the Hessian matrix. To be able to exploit Hessian sparsity, existing optimization approach…
We present a MATLAB implementation of the symmetric rank-one (SC-SR1) method that solves trust-region. subproblems when a limited-memory symmetric rank-one (L-SR1) matrix is used in place of the true Hessian matrix, which can be used for large-scale optimization. The method takes advantage of two shape-changing norms[Burdakov and Yuan 2002; Burdakov et al. 2017] to decompose the trust-region subproblem into two separate problems. Using one of the proposed norms, the resulting subproblems have closed-form solutions. Meanwhile, using the other proposed norm, one of the resulting subproblems has …
In this paper, we modify the adaptive cubic regularization method for large-scale unconstrained optimization problem by using a real positive definite scalar matrix to approximate the exact Hessian. Combining with the nonmonotone technique, we also give a variant of the modified algorithm. Under some reasonable conditions, we analyze the global convergence of the proposed methods. Numerical experiments are performed and the obtained results show satisfactory performance when compared to the standard trust region method, adaptive regularization algorithm with cubics and the simple model trust-r…
For solving large-scale non-convex problems, we propose inexact variants of trust region and adaptive cubic regularization methods, which, to increase efficiency, incorporate various approximations. In particular, in addition to approximate sub-problem solves, both the Hessian and the gradient are suitably approximated. Using rather mild conditions on such approximations, we show that our proposed inexact methods achieve similar optimal worst-case iteration complexities as the exact counterparts. Our proposed algorithms, and their respective theoretical analysis, do not require knowledge of an…
We propose a novel framework for analyzing convergence rates of stochastic optimization algorithms with adaptive step sizes. This framework is based on analyzing properties of an underlying generic stochastic process, in particular by deriving a bound on the expected stopping time of this process. We utilize this framework to analyze the bounds on expected global convergence rates of a stochastic variant of a traditional trust region method, introduced in \cite{ChenMenickellyScheinberg2014}. While traditional trust region methods rely on exact computations of the gradient, Hessian and values o…
BUP-TR: Bayesian Underdetermined Projection Trust-Region Methods for Derivative-Free Optimization
Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Lawrence Berkeley National Laboratory（劳伦斯伯克利国家实验室）
A primal-dual interior point trust region method for second-order stationary points of Riemannian inequality-constrained optimization problems
Graduate School of Information Science and Technology, The University of Tokyo（东京大学大学院信息理工学系研究科）；Center for Advanced Intelligence Project, RIKEN（理化学研究所先进智能项目中心）；Faculty of Science and Technology, Seikei University（立教大学理学部）
Stochastic Zeroth-Order Optimization under Strongly Convexity and Lipschitz Hessian: Minimax Sample Complexity
University of California, Santa Barbara（加利福尼亚大学圣巴巴拉分校）；University of Texas at Dallas（德克萨斯大学达拉斯分校）；University of California, Berkeley（加利福尼亚大学伯克利分校）；New York University（纽约大学）；Princeton University（普林斯顿大学）
Adaptive regularization with cubics (ARC) is an algorithm for unconstrained, non-convex optimization. Akin to the popular trust-region method, its iterations can be thought of as approximate, safe-guarded Newton steps. For cost functions with Lipschitz continuous Hessian, ARC has optimal iteration complexity, in the sense that it produces an iterate with gradient smaller than $\varepsilon$ in $O(1/\varepsilon^{1.5})$ iterations. For the same price, it can also guarantee a Hessian with smallest eigenvalue larger than $-\varepsilon^{1/2}$. In this paper, we study a generalization of ARC to optim…
The Trust Region Subproblem is a fundamental optimization problem that takes a pivotal role in Trust Region Methods. However, the problem, and variants of it, also arise in quite a few other applications. In this article, we present a family of iterative Riemannian optimization algorithms for a variant of the Trust Region Subproblem that replaces the inequality constraint with an equality constraint, and converge to a global optimum. Our approach uses either a trivial or a non-trivial Riemannian geometry of the search-space, and requires only minimal spectral information about the quadratic co…
This paper considers an explicit continuation method and the trust-region updating strategy for the unconstrained optimization problem. Moreover, in order to improve its computational efficiency and robustness, the new method uses the switching preconditioning technique. In the well-conditioned phase, the new method uses the L-BFGS method as the preconditioning technique in order to improve its computational efficiency. Otherwise, the new method uses the inverse of the Hessian matrix as the pre-conditioner in order to improve its robustness. Numerical results aslo show that the new method is m…
Gaussian Mixture Models are a powerful tool in Data Science and Statistics that are mainly used for clustering and density approximation. The task of estimating the model parameters is in practice often solved by the Expectation Maximization (EM) algorithm which has its benefits in its simplicity and low per-iteration costs. However, the EM converges slowly if there is a large share of hidden information or overlapping clusters. Recent advances in manifold optimization for Gaussian Mixture Models have gained increasing interest. We introduce an explicit formula for the Riemannian Hessian for G…
TRSVR: An Adaptive Stochastic Trust-Region Method with Variance Reduction
Department of Mathematics, University of California, Berkeley, CA, USA（加州大学伯克利分校数学系）；Boston University, MA, USA（波士顿大学）；Department of IEOR, University of California, Berkeley, CA, USA（加州大学伯克利分校工业工程与运筹学系）
In this contribution we device and analyze improved variants of the non-conforming dual approach for trust-region reduced basis (TR-RB) approximation of PDE-constrained parameter optimization that has recently been introduced in [Keil et al.. A non-conforming dual approach for adaptive Trust-Region Reduced Basis approximation of PDE-constrained optimization. arXiv:2006.09297, 2020]. The proposed methods use model order reduction techniques for parametrized PDEs to significantly reduce the computational demand of parameter optimization with PDE constraints in the context of large-scale or multi…
An algorithm for solving nonconvex smooth optimization problems is proposed, analyzed, and tested. The algorithm is an extension of the Trust Region Algorithm with Contractions and Expansions (TRACE) [Math. Prog. 162(1):132, 2017]. In particular, the extension allows the algorithm to use inexact solutions of the arising subproblems, which is an important feature for solving large-scale problems. Inexactness is allowed in a manner such that the optimal iteration complexity of ${\cal O}(ε^{-3/2})$ for attaining an $ε$-approximate first-order stationary point is maintained while the worst-case co…
We establish or refute the optimality of inexact second-order methods for unconstrained nonconvex optimization from the point of view of worst-case evaluation complexity, improving and generalizing the results of Cartis, Gould and Toint (2010,2011). To this aim, we consider a new general class of inexact second-order algorithms for unconstrained optimization that includes regularization and trust-region variations of Newton's method as well as of their linesearch variants. For each method in this class and arbitrary accuracy threshold $ε\in (0,1)$, we exhibit a smooth objective function with b…
