For optimization problems with nonlinear constraints, linearly constrained Lagrangian (LCL) methods sequentially minimize a Lagrangian function subject to linearized constraints. These methods converge rapidly near a solution but may not be reliable from arbitrary starting points. The well known example \MINOS\ has proven effective on many large problems. Its success motivates us to propose a globally convergent variant. Our stabilized LCL method possesses two important properties: the subproblems are always feasible, and they may be solved inexactly. These features are present in \MINOS only as heuristics.
The new algorithm has been implemented in \Matlab, with the option to use either the \MINOS or \SNOPT Fortran codes to solve the linearly constrained subproblems. Only first derivatives are required. We present numerical results on a nonlinear subset of the \COPS, \CUTE, and HS test-problem sets, which include many large examples. The results demonstrate the robustness and efficiency of the stabilized LCL procedure.
We study the \emph{Proximal Alternating Predictor-Corrector} (PAPC) algorithm introduced recently by Drori, Sabach and Teboulle to solve nonsmooth structured convex-concave saddle point problems consisting of the sum of a smooth convex function, a finite collection of nonsmooth convex functions and bilinear terms. We introduce the notion of pointwise quadratic supportability, which is a relaxation of a standard strong convexity assumption and allows us to show that the primal sequence is R-linearly convergent to an optimal solution and the primal-dual sequence is globally Q-linearly convergent…
We consider the application of the type-I Anderson acceleration to solving general non-smooth fixed-point problems. By interleaving with safe-guarding steps, and employing a Powell-type regularization and a re-start checking for strong linear independence of the updates, we propose the first globally convergent variant of Anderson acceleration assuming only that the fixed-point iteration is non-expansive. We show by extensive numerical experiments that many first order algorithms can be improved, especially in their terminal convergence, with the proposed algorithm. Our proposed method of acce…
In this paper, a new conjugate gradient-like algorithm is proposed to solve unconstrained optimization problems. The step directions generated by the new algorithm satisfy sufficient descent condition independent of the line search. The global convergence of the new algorithm, with the Armijo backtracking line search, is proved. Numerical experiments indicate the efficiency and robustness of the new algorithm.
We consider the efficient minimization of a nonlinear, strictly convex functional with $\ell_1$-penalty term. Such minimization problems appear in a wide range of applications like Tikhonov regularization of (non)linear inverse problems with sparsity constraints. In (2015 Inverse Problems (31) 025005), a globalized Bouligand-semismooth Newton method was presented for $\ell_1$-Tikhonov regularization of linear inverse problems. Nevertheless, a technical assumption on the accumulation point of the sequence of iterates was necessary to prove global convergence. Here, we generalize this method to …
Motivated by machine learning problems over large data sets and distributed optimization over networks, we develop and analyze a new method called incremental Newton method for minimizing the sum of a large number of strongly convex functions. We show that our method is globally convergent for a variable stepsize rule. We further show that under a gradient growth condition, convergence rate is linear for both variable and constant stepsize rules. By means of an example, we show that without the gradient growth condition, incremental Newton method cannot achieve linear convergence. Our analysis…
In this paper, an efficient modified Newton type algorithm is proposed for nonlinear unconstrianed optimization problems. The modified Hessian is a convex combination of the identity matrix (for steepest descent algorithm) and the Hessian matrix (for Newton algorithm). The coefficients of the convex combination are dynamically chosen in every iteration. The algorithm is proved to be globally and quadratically convergent for (convex and nonconvex) nonlinear functions. Efficient implementation is described. Numerical test on widely used CUTE test problems is conducted for the new algorithm. The …
This paper addresses some trust-region methods equipped with nonmonotone strategies for solving nonlinear unconstrained optimization problems. More specifically, the importance of using nonmonotone techniques in nonlinear optimization is motivated, then two new nonmonotone terms are proposed, and their combinations into the traditional trust-region framework are studied. The global convergence to first- and second-order stationary points and local superlinear and quadratic convergence rates for both algorithms are established. Numerical experiments on the \textsf{CUTEst} test collection of unc…
We propose a globally convergent Gauss-Newton algorithm for finding a local optimal solution of a non-convex and possibly non-smooth optimization problem. The algorithm that we present is based on a Gauss-Newton-type iteration for the non-smooth penalized formulation of the original problem. We establish a global convergence rate for this scheme from any initial point to a stationary point of the problem while using an exact penalty formulation. Under some more restrictive conditions we also derive local quadratic convergence for this scheme. We apply our proposed algorithm to solve the Altern…
This article deals with the conjugate gradient method on a Riemannian manifold with interest in global convergence analysis. The existing conjugate gradient algorithms on a manifold endowed with a vector transport need the assumption that the vector transport does not increase the norm of tangent vectors, in order to confirm that generated sequences have a global convergence property. In this article, the notion of a scaled vector transport is introduced to improve the algorithm so that the generated sequences may have a global convergence property under a relaxed assumption. In the proposed a…
The paper proposes and justifies a new algorithm of the proximal Newton type to solve a broad class of nonsmooth composite convex optimization problems without strong convexity assumptions. Based on advanced notions and techniques of variational analysis, we establish implementable results on the global convergence of the proposed algorithm as well as its local convergence with superlinear and quadratic rates. For certain structured problems, the obtained local convergence conditions do not require the local Lipschitz continuity of the corresponding Hessian mappings that is a crucial assumptio…
In this paper we are interested in the problem of state observation of state-affine nonlinear systems. Our main contribution is to propose a globally exponentially convergent observer that requires only the necessary assumption of observability of the system. To the best of the authors' knowledge this is the first time such a result is reported in the literature.
Motivated by the increasing availability of high-performance parallel computing, we design a distributed parallel algorithm for linearly-coupled block-structured nonconvex constrained optimization problems. Our algorithm performs Jacobi-type proximal updates of the augmented Lagrangian function, requiring only local solutions of separable block nonlinear programming (NLP) problems. We provide a cheap and explicitly computable Lyapunov function that allows us to establish global and local sublinear convergence of our algorithm, its iteration complexity, as well as simple, practical and theoreti…
Semidefinite programs (SDP) are one of the most versatile frameworks in numerical optimization, serving as generalizations of many conic programs and as relaxations of NP-hard combinatorial problems. Their main drawback is their computational and memory complexity, which sets a practical limit to the size of problems solvable by off-the-shelf SDP solvers. To circumvent this fact, many algorithms have been proposed to exploit the structure of particular problems and increase the scalability of SDPs for those problem instances. Progress has been less steep, however, for general-case SDPs. In thi…
We propose globally exponentially convergent continuous observers for invariant kinematic systems on finite-dimensional matrix Lie groups. Such an observer estimates, from measurements of landmarks, vectors and biased velocity, both the system state and the unknown constant bias in velocity measurement, where the state belongs to the state-space Lie group and the velocity to the Lie algebra of the Lie group. The main technique is to embed a given system defined on a matrix Lie group into Euclidean space and build observers in the Euclidean space. The theory is illustrated with the special Eucl…
We propose a fast and robust scheme for the direct minimization of the Ohta-Kawasaki energy that characterizes the microphase separation of diblock copolymer melts. The scheme employs a globally convergent modified Newton method with line search which is shown to be mass-conservative, energy-descending, asymptotically quadratically convergent, and three orders of magnitude more efficient than the commonly-used gradient flow approach. The regularity and the first-order condition of minimizers are analyzed. A numerical study of the chemical substrate guided directed self-assembly of diblock copo…
In this paper, we propose a globally convergent method for solving constrained nonlinear systems. The method combines an efficient Newton conditional gradient method with a derivative-free and nonmonotone linesearch strategy. The global convergence analysis of the proposed method is established under suitable conditions, and some preliminary numerical experiments are given to illustrate its performance.
We present a globally convergent method to accelerate density-based topology optimization using projection-based reduced-order models (ROMs) and trust-region methods. To accelerate topology optimization, we replace the large-scale finite element simulation, which dominates the computational cost, with ROMs that reduce the cost of objective function and gradient evaluations by orders of magnitude. To guarantee convergence, we first introduce a trust-region method that employs generalized trust-region constraints and prove it is globally convergent. We then devise a class of globally convergent …
School of Science, Sun Yat-sen University（中山大学理学院）；Department of Systems Engineering and Engineering Management, The Chinese University of Hong Kong（香港中文大学系统工程及工程管理学系）；School of Mathematical Sciences, Xiamen University（厦门大学数学科学学院）
We present two globally convergent Levenberg-Marquardt methods for finding zeros of Hölder metrically subregular mappings that may have non-isolated zeros. The first method unifies the Levenberg- Marquardt direction and an Armijo-type line search, while the second incorporates this direction with a nonmonotone trust-region technique. For both methods, we prove the global convergence to a first-order stationary point of the associated merit function. Furthermore, the worst-case global complexity of these methods are provided, indicating that an approximate stationary point can be computed in at…
The paper proposes and develops new globally convergent algorithms of the generalized damped Newton type for solving important classes of nonsmooth optimization problems. These algorithms are based on the theory and calculations of second-order subdifferentials of nonsmooth functions with employing the machinery of second-order variational analysis and generalized differentiation. First we develop a globally superlinearly convergent damped Newton-type algorithm for the class of continuously differentiable functions with Lipschitzian gradients, which are nonsmooth of second order. Then we desig…
Università della Svizzera italiana（瑞士意大利语大学）；Euler Institute（欧拉研究所）
We propose a globally convergent multilevel training method for deep residual networks (ResNets). The devised method can be seen as a novel variant of the recursive multilevel trust-region (RMTR) method, which operates in hybrid (stochastic-deterministic) settings by adaptively adjusting mini-batch sizes during the training. The multilevel hierarchy and the transfer operators are constructed by exploiting a dynamical system's viewpoint, which interprets forward propagation through the ResNet as a forward Euler discretization of an initial value problem. In contrast to traditional training appr…
Nonconvex optimization problems arise in many areas of computational science and engineering and are (approximately) solved by a variety of algorithms. Existing algorithms usually only have local convergence or subsequence convergence of their iterates. We propose an algorithm for a generic nonconvex optimization formulation, establish the convergence of its whole iterate sequence to a critical point along with a rate of convergence, and numerically demonstrate its efficiency. Specially, we consider the problem of minimizing a nonconvex objective function. Its variables can be treated as one b…
A globally convergent Carleman-Picard method for an inverse initial-value problem for a nonlinear diffusive coagulation-fragmentation equation coagulation-fragmentation equation
California State University, Long Beach（加州州立大学长滩分校）；Texas A&M University（德克萨斯农工大学）；University of North Carolina at Charlotte（北卡罗来纳大学夏洛特分校）
Stochastic variance reduced optimization methods are known to be globally convergent while they suffer from slow local convergence, especially when moderate or high accuracy is needed. To alleviate this problem, we propose an optimization algorithm -- which we refer to as a hybrid acceleration scheme -- for a class of proximal variance reduced stochastic optimization algorithms. The proposed optimization scheme combines a fast locally convergent algorithm, such as a quasi--Newton method, with a globally convergent variance reduced stochastic algorithm, for instance SAGA or L--SVRG. Our global …
Polytechnique Montréal（蒙特利尔综合理工学院）；University of Basel（巴塞尔大学）；Johannes Kepler University Linz（林茨约翰·开普勒大学）
Evolutionary strategies have recently been shown to achieve competing levels of performance for complex optimization problems in reinforcement learning. In such problems, one often needs to optimize an objective function subject to a set of constraints, including for instance constraints on the entropy of a policy or to restrict the possible set of actions or states accessible to an agent. Convergence guarantees for evolutionary strategies to optimize stochastic constrained problems are however lacking in the literature. In this work, we address this problem by designing a novel optimization a…
Convex-composite optimization, which minimizes an objective function represented by the sum of a differentiable function and a convex one, is widely used in machine learning and signal/image processing. Fast Iterative Shrinkage Thresholding Algorithm (FISTA) is a typical method for solving this problem and has a global convergence rate of $O(1 / k^2)$. Recently, this has been extended to multi-objective optimization, together with the proof of the $O(1 / k^2)$ global convergence rate. However, its momentum factor is classical, and the convergence of its iterates has not been proven. In this wo…
INRIA（法国国家信息与自动化研究所）；Département d’Informatique de l’École Normale Supérieure（巴黎高等师范学院计算机系）；PSL Research University（巴黎文理研究大学）
In this paper, we study large-scale convex optimization algorithms based on the Newton method applied to regularized generalized self-concordant losses, which include logistic regression and softmax regression. We first prove that our new simple scheme based on a sequence of problems with decreasing regularization parameters is provably globally convergent, that this convergence is linear with a constant factor which scales only logarithmically with the condition number. In the parametric setting, we obtain an algorithm with the same scaling than regular first-order methods but with an improve…
MIT（麻省理工学院）；University of California, Berkeley（加州大学伯克利分校）
We introduce the first direct policy search algorithm which provably converges to the globally optimal $\textit{dynamic}$ filter for the classical problem of predicting the outputs of a linear dynamical system, given noisy, partial observations. Despite the ubiquity of partial observability in practice, theoretical guarantees for direct policy search algorithms, one of the backbones of modern reinforcement learning, have proven difficult to achieve. This is primarily due to the degeneracies which arise when optimizing over filters that maintain internal state. In this paper, we provide a new p…
IIT Kanpur（印度理工学院坎普尔分校）；Microsoft Research India（微软印度研究院）
We provide the first global model recovery results for the IRLS (iteratively reweighted least squares) heuristic for robust regression problems. IRLS is known to offer excellent performance, despite bad initializations and data corruption, for several parameter estimation problems. Existing analyses of IRLS frequently require careful initialization, thus offering only local convergence guarantees. We remedy this by proposing augmentations to the basic IRLS routine that not only offer guaranteed global recovery, but in practice also outperform state-of-the-art algorithms for robust regression. …
This paper aims to study a majorized alternating direction method of multipliers with indefinite proximal terms (iPADMM) for convex composite optimization problems. We show that the majorized iPADMM for 2-block convex optimization problems converges globally under weaker conditions than those used in the literature and exhibits a linear convergence rate under a local error bound condition. Based on these, we establish the linear rate convergence results for a symmetric Gaussian-Seidel based majorized iPADMM, which is designed for multi-block composite convex optimization problems. Moreover, we…
We consider the problem of computing a Wasserstein barycenter for a set of discrete probability distributions with finite supports, which finds many applications in areas such as statistics, machine learning and image processing. When the support points of the barycenter are pre-specified, this problem can be modeled as a linear programming (LP) problem whose size can be extremely large. To handle this large-scale LP, we analyse the structure of its dual problem, which is conceivably more tractable and can be reformulated as a well-structured convex problem with 3 kinds of block variables and …
Following Demidovich's concept and definition of convergent systems, we analyze the optimal nonlinear damping control, recently proposed [1] for the second-order systems. Targeting the problem of output regulation, correspondingly tracking of $\mathcal{C}^1$-trajectories, it is shown that all solutions of the control system are globally uniformly asymptotically stable. The existence of the unique limit solution in the origin of the control error and its time derivative coordinates are shown in the sense of Demidovich's convergent dynamics. Explanative numerical examples are also provided along…
The minimum volume enclosing ellipsoid (MVEE) problem is an optimization problem in the basis of many practical problems. This paper describes some new properties of this model and proposes a first-order oracle algorithm, the Adjusted Coordinate Descent (ACD) algorithm, to address the MVEE problem. The ACD algorithm is globally linear convergent and has an overwhelming advantage over the other algorithms in cases where the dimension of the data is large. Moreover, as a byproduct of the convergence property of the ACD algorithm, we prove the global linear convergence of the Frank-Wolfe type alg…
This paper studies the long-existing idea of adding a nice smooth function to "smooth" a non-differentiable objective function in the context of sparse optimization, in particular, the minimization of $||x||_1+1/(2α)||x||_2^2$, where $x$ is a vector, as well as the minimization of $||X||_*+1/(2α)||X||_F^2$, where $X$ is a matrix and $||X||_*$ and $||X||_F$ are the nuclear and Frobenius norms of $X$, respectively. We show that they can efficiently recover sparse vectors and low-rank matrices. In particular, they enjoy exact and stable recovery guarantees similar to those known for minimizing $|…
International Computer Science Institute（国际计算机科学研究所）；University of California at Berkeley（加州大学伯克利分校）
Large scale optimization problems are ubiquitous in machine learning and data analysis and there is a plethora of algorithms for solving such problems. Many of these algorithms employ sub-sampling, as a way to either speed up the computations and/or to implicitly implement a form of statistical regularization. In this paper, we consider second-order iterative optimization algorithms and we provide bounds on the convergence of the variants of Newton's method that incorporate uniform sub-sampling as a means to estimate the gradient and/or Hessian. Our bounds are non-asymptotic and quantitative. …
An efficient, globally convergent method for optimization under uncertainty using adaptive model reduction and sparse grids
Department of Mathematics, Lawrence Berkeley National Laboratory, University of California, Berkeley（加州大学伯克利分校劳伦斯伯克利国家实验室数学系）；University of California, Berkeley（加州大学伯克利分校）；Department of Aerospace and Mechanical Engineering, University of Notre Dame（圣母院大学航空航天与机械工程系）；University of Notre Dame（圣母院大学）；Sandia National Laboratories（桑迪亚国家实验室）
This paper presents the SCvx algorithm, a successive convexification algorithm designed to solve non-convex constrained optimal control problems with global convergence and superlinear convergence-rate guarantees. The proposed algorithm can handle nonlinear dynamics and non-convex state and control constraints. It solves the original problem to optimality by successively linearizing non-convex dynamics and constraints about the solution of the previous iteration. The resulting convex subproblems are numerically tractable, and can be computed quickly and reliably using convex optimization solve…
