The idea of a finite collection of closed sets having "strongly regular intersection" at a given point is crucial in variational analysis. We show that this central theoretical tool also has striking algorithmic consequences. Specifically, we consider the case of two sets, one of which we assume to be suitably "regular" (special cases being convex sets, smooth manifolds, or feasible regions satisfying the Mangasarian-Fromovitz constraint qualification). We then prove that von Neumann's method of "alternating projections" converges locally to a point in the intersection, at a linear rate associated with a modulus of regularity. As a consequence, in the case of several arbitrary closed sets having strongly regular intersection at some point, the method of "averaged projections" converges locally at a linear rate to a point in the intersection. Inexact versions of both algorithms also converge linearly.
For some typical and widely used non-convex half-quadratic regularization models and the Ambrosio-Tortorelli approximate Mumford-Shah model, based on the Kurdyka-Łojasiewicz analysis and the recent nonconvex proximal algorithms, we developed an efficient preconditioned framework aiming at the linear subproblems that appeared in the nonlinear alternating minimization procedure. Solving large-scale linear subproblems is always important and challenging for lots of alternating minimization algorithms. By cooperating the efficient and classical preconditioned iterations into the nonlinear and nonc…
We propose two numerical algorithms in the fully nonconvex setting for the minimization of the sum of a smooth function and the composition of a nonsmooth function with a linear operator. The iterative schemes are formulated in the spirit of the proximal alternating direction method of multipliers and its linearized variant, respectively. The proximal terms are introduced via variable metrics, a fact which allows us to derive new proximal splitting algorithms for nonconvex structured optimization problems, as particular instances of the general schemes. Under mild conditions on the sequence of…
This work investigates the theoretical performance of the alternating-direction method of multipliers (ADMM) as it applies to nonconvex optimization problems, and in particular, problems with nonconvex constraint sets. The alternating direction method of multipliers is an optimization method that has largely been analyzed for convex problems. The ultimate goal is to assess what kind of theoretical convergence properties the method has in the nonconvex case, and to this end, theoretical contributions are two-fold. First, this work analyzes the method with local solution of the ADMM subproblems,…
Iteratively Linearized Reweighted Alternating Direction Method of Multipliers for a Class of Nonconvex Problems
Department of Mathematics, National University of Defense Technology（国防科技大学数学系）；College of Computer, National University of Defense Technology（国防科技大学计算机学院）；The State Key Laboratory for High Performance Computation, National University of Defense Technology（国防科技大学高性能计算国家重点实验室）；Hunan Key Laboratory for Computation and Simulation in Science and Engineering, School of Mathematics and Computational Science, Xiangtan University（湖南计算与模拟科学工程重点实验室，湘潭大学数学与计算科学学院）
The classic Alternating Direction Method of Multipliers (ADMM) is a popular framework to solve linear-equality constrained problems. In this paper, we extend the ADMM naturally to nonlinear equality-constrained problems, called neADMM. The difficulty of neADMM is to solve nonconvex subproblems. We provide globally optimal solutions to them in two important applications. Experiments on synthetic and real-world datasets demonstrate excellent performance and scalability of our proposed neADMM over existing state-of-the-start methods.
In this paper we propose a proximal algorithm for minimizing an objective function of two block variables consisting of three terms: 1) a smooth function, 2) a nonsmooth function which is a composition between a strictly increasing, concave, differentiable function and a convex nonsmooth function, and 3) a smooth function which couples the two block variables. We propose a variable metric composite proximal alternating linearized minimization (CPALM) to solve this class of problems. Building on the powerful Kurdyka-Łojasiewicz property, we derive the convergence analysis and establish that eac…
Alternating gradient-descent-ascent (AltGDA) is an optimization algorithm that has been widely used for model training in various machine learning applications, which aims to solve a nonconvex minimax optimization problem. However, the existing studies show that it suffers from a high computation complexity in nonconvex minimax optimization. In this paper, we develop a single-loop and fast AltGDA-type algorithm that leverages proximal gradient updates and momentum acceleration to solve regularized nonconvex minimax optimization problems. By leveraging the momentum acceleration technique, we pr…
We study several versions of the alternating direction method of multipliers (ADMM) for solving the convex problem of finding the distance between two ellipsoids and the nonconvex problem of finding the distance between the boundaries of two ellipsoids. In the convex case we present the ADMM with and without automatic penalty updates and demonstrate via numerical experiments on problems of various dimensions that our methods significantly outperform all other existing methods for finding the distance between ellipsoids. In the nonconvex case we propose a heuristic rule for updating the penalty…
Shanghai University（上海大学）；Georgia Institute of Technology（佐治亚理工学院）
Much recent research effort has been directed to the development of efficient algorithms for solving minimax problems with theoretical convergence guarantees due to the relevance of these problems to a few emergent applications. In this paper, we propose a unified single-loop alternating gradient projection (AGP) algorithm for solving smooth nonconvex-(strongly) concave and (strongly) convex-nonconcave minimax problems. AGP employs simple gradient projection steps for updating the primal and dual variables alternatively at each iteration. We show that it can find an $\varepsilon$-stationary po…
Derivative-free Alternating Projection Algorithms for General Nonconvex-Concave Minimax Problems
Shanghai University（上海大学）；Newtouch Center for Mathematics of Shanghai University（上海大学新触数学中心）；Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院）
A Riemannian Alternating Descent Ascent Algorithmic Framework for Nonconvex-Linear Minimax Problems on Riemannian Manifolds
ICMSEC, Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Ministry of Education Key Laboratory of NSLSCS, School of Mathematical Sciences, Nanjing Normal University（南京师范大学数学科学学院）；Ministry of Education Key Laboratory of Mathematics and Information Networks, School of Mathematical Sciences, Beijing University of Posts and Telecommunications（北京邮电大学数学科学学院）；Department of Systems Engineering and Engineering Management, The Chinese University of Hong Kong（香港中文大学系统工程与工程管理学系）
Consider the minimization of a nonconvex differentiable function over a polyhedron. A popular primal-dual first-order method for this problem is to perform a gradient projection iteration for the augmented Lagrangian function and then update the dual multiplier vector using the constraint residual. However, numerical examples show that this approach can exhibit "oscillation" and may not converge. In this paper, we propose a proximal alternating direction method of multipliers for the multi-block version of this problem. A distinctive feature of this method is the introduction of a "smoothed" (…
Zeroth-Order Alternating Gradient Descent Ascent Algorithms for a Class of Nonconvex-Nonconcave Minimax Problems
Shanghai University（上海大学）；Newtouch Center for Mathematics of Shanghai University（上海大学纽崔克数学中心）；Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院）
Alternating Iteratively Reweighted $\ell_1$ and Subspace Newton Algorithms for Nonconvex Sparse Optimization
School of Information Science and Technology, ShanghaiTech University（上海科技大学信息科学与技术学院）；School of Mathematics and Statistics, Henan University（河南大学数学与统计学院）；Center for Applied Mathematics of Henan Province, Henan University（河南大学河南省应用数学中心）
Zeroth-Order primal-dual Alternating Projection Gradient Algorithms for Nonconvex Minimax Problems with Coupled linear Constraints
Department of Mathematics, College of Sciences, Shanghai University（上海大学数学系）；Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院）；Newtouch Center for Mathematics of Shanghai University（上海大学数学中心）；State Key Laboratory of Mathematical Sciences, Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院数学科学国家重点实验室）；School of Mathematical Sciences, University of Chinese Academy of Sciences（中国科学院大学数学科学学院）
University of Pittsburgh（匹兹堡大学）；Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；MIIT Key Laboratory of Pattern Analysis & Machine Intelligence（工业和信息化部模式分析与机器智能重点实验室）；JD Finance America Corporation（京东金融美国公司）
In this paper, we propose a faster stochastic alternating direction method of multipliers (ADMM) for nonconvex optimization by using a new stochastic path-integrated differential estimator (SPIDER), called as SPIDER-ADMM. Moreover, we prove that the SPIDER-ADMM achieves a record-breaking incremental first-order oracle (IFO) complexity of $\mathcal{O}(n+n^{1/2}ε^{-1})$ for finding an $ε$-approximate stationary point, which improves the deterministic ADMM by a factor $\mathcal{O}(n^{1/2})$, where $n$ denotes the sample size. As one of major contribution of this paper, we provide a new theoretica…
Phase retrieval can be expressed as one non-convex constrained optimization problem to identify one phase minimizer in the primal space. Many iterative transform techniques have been proposed to identify the minimizer, e.g., relaxed averaged alternating reflections(RAAR) algorithms. RAAR algorithm is one alternating direction method of multipliers(ADMM) with one penalty parameter. Pairing with multipliers (dual vectors), phase vectors are lifted to higher dimensional vectors, RAAR algorithms actually is one continuation algorithm, which searches for local Nash equilibria in a primal-dual space…
Huawei Belgium Research Center（华为比利时研究中心）；Carnegie Mellon University（卡内基梅隆大学）；Université de Mons（蒙斯大学）
In this paper, we propose an algorithmic framework, dubbed inertial alternating direction methods of multipliers (iADMM), for solving a class of nonconvex nonsmooth multiblock composite optimization problems with linear constraints. Our framework employs the general minimization-majorization (MM) principle to update each block of variables so as to not only unify the convergence analysis of previous ADMM that use specific surrogate functions in the MM step, but also lead to new efficient ADMM schemes. To the best of our knowledge, in the nonconvex nonsmooth setting, ADMM used in combination wi…
An optimization algorithm for a group of nonsmooth nonconvex problems inspired by two-stage stochastic programming problems is proposed. The main challenges for these problems include (1) the problems lack the popular lower-type properties such as prox-regularity assumed in many nonsmooth nonconvex optimization algorithms, (2) the objective can not be analytically expressed and (3) the evaluation of function values and subgradients are computationally expensive. To address these challenges, this report first examines the properties that exist in many two-stage problems, specifically upper-C^2 …
In this paper, we consider a proximal linearized alternating direction method of multipliers (PL-ADMM) for solving linearly constrained nonconvex and possibly nonsmooth optimization problems. The algorithm is generalized by using variable metric proximal terms in the primal updates and an over-relaxation stepsize in the multiplier update. We prove that the sequence generated by this method is bounded and its limit points are critical points. Under the powerful Kurdyka-{Łojasiewicz} properties we prove that the sequence has a finite length thus converges, and we drive its convergence rates.
