本文针对光滑非凸线性约束最优化问题，提出了一种新的分解方法--邻近原始对偶法（Prox-PDA）.所提出的方法是原始-对偶的基础上，其中的原始步骤最小化某些近似的增广拉格朗日的问题，和对偶步骤进行近似的对偶上升。在初始步骤中使用的近似能够分解变量块，使得可以通过利用问题结构来获得简单的子问题。理论上，我们证明了当增广拉格朗日量中的罚参数大于给定阈值时，Prox-PDA全局和次线性地收敛到稳态解集（即，在$\mathcal{O}(1/r)$的速率中，稳定性的某种度量降低，其中$r$是迭代计数器）。有趣的是，当将Prox-PDA的变体应用于分布式非凸优化问题（在连通的无向图上）时，所得算法与流行的EXTRA算法一致[Shi et al 2014]，该算法仅在凸情况下工作。我们的分析表明，EXTRA及其变种全局次线性收敛到某些非凸分布最优化问题的平稳解。Prox-PDA有许多可能的扩展，我们提出了一个特定的扩展到某些非凸分布矩阵分解问题。
英文摘要：
In this paper, we propose a new decomposition approach named the proximal primal dual algorithm (Prox-PDA) for smooth nonconvex linearly constrained optimization problems. The proposed approach is primal-dual based, where the primal step minimizes certain approximation of the augmented Lagrangian of the problem, and the dual step performs an approximate dual ascent. The approximation used in the primal step is able to decompose the variable blocks, making it possible to obtain simple subproblems by leveraging the problem structures. Theoretically, we show that whenever the penalty parameter in the augmented Lagrangian is larger than a given threshold, the Prox-PDA converges to the set of stationary solutions, globally and in a sublinear manner (i.e., certain measure of stationarity decreases in the rate of $\mathcal{O}(1/r)$, where $r$ is the iteration counter). Interestingly, when applying a variant of the Prox-PDA to the problem of distributed nonconvex optimization (over a connected undirected graph), the resulting algorithm coincides with the popular EXTRA algorithm [Shi et al 2014], which is only known to work in convex cases. Our analysis implies that EXTRA and its variants converge globally sublinearly to stationary solutions of certain nonconvex distributed optimization problem. There are many possible extensions of the Prox-PDA, and we present one particular extension to certain nonconvex distributed matrix factorization problem.
We provide an overview of primal-dual algorithms for nonsmooth and non-convex-concave saddle-point problems. This flows around a new analysis of such methods, using Bregman divergences to formulate simplified conditions for convergence.
Nonlinearly constrained nonconvex and nonsmooth optimization models play an increasingly important role in machine learning, statistics and data analytics. In this paper, based on the augmented Lagrangian function we introduce a flexible first-order primal-dual method, to be called nonconvex auxiliary problem principle of augmented Lagrangian (NAPP-AL), for solving a class of nonlinearly constrained nonconvex and nonsmooth optimization problems. We demonstrate that NAPP-AL converges to a stationary solution at the rate of o(1/\sqrt{k}), where k is the number of iterations. Moreover, under an a…
On the Regularization of Primal-Dual Dynamics in Time-Dependent Nonconvex Optimization
School of Science and Technology, IE University（IE大学科学与技术学院）；Department of Mechanical and Aerospace Engineering, University of California San Diego（加州大学圣地亚哥分校机械与航空航天工程系）
The distributed nonconvex optimization problem of minimizing a global cost function formed by a sum of $n$ local cost functions by using local information exchange is considered. This problem is an important component of many machine learning techniques with data parallelism, such as deep learning and federated learning. We propose a distributed primal--dual stochastic gradient descent (SGD) algorithm, suitable for arbitrarily connected communication networks and any smooth (possibly nonconvex) cost functions. We show that the proposed algorithm achieves the linear speedup convergence rate $\m…
This paper considers a nonconvex optimization problem that evolves over time, and addresses the synthesis and analysis of regularized primal-dual gradient methods to track a Karush-Kuhn-Tucker (KKT) trajectory. The proposed regularized primal-dual gradient methods are implemented in a running fashion, in the sense that the underlying optimization problem changes during the iterations of the algorithms. For a problem with twice continuously differentiable cost and constraints, and under a generalization of the Mangasarian-Fromovitz constraint qualification, sufficient conditions are derived for…
This paper considers the distributed nonconvex optimization problem of minimizing a global cost function formed by a sum of local cost functions by using local information exchange. We first consider a distributed first-order primal-dual algorithm. We show that it converges sublinearly to a stationary point if each local cost function is smooth and linearly to a global optimum under an additional condition that the global cost function satisfies the Polyak-Łojasiewicz condition. This condition is weaker than strong convexity, which is a standard condition for proving linear convergence of dist…
In this paper, we consider the problem of recovering a sparse signal based on penalized least squares formulations. We develop a novel algorithm of primal-dual active set type for a class of nonconvex sparsity-promoting penalties, including $\ell^0$, bridge, smoothly clipped absolute deviation, capped $\ell^1$ and minimax concavity penalty. First we establish the existence of a global minimizer for the related optimization problems. Then we derive a novel necessary optimality condition for the global minimizer using the associated thresholding operator. The solutions to the optimality system a…
A Unifying Primal-Dual Proximal Framework for Distributed Nonconvex Optimization
School of Information Science and Technology, Shanghaitech University（上海科技大学信息科学与技术学院）；Shanghai Engineering Research Center of Energy Efficient and Custon AI IC（上海市能效与定制人工智能集成电路工程研究中心）
Zeroth-Order primal-dual Alternating Projection Gradient Algorithms for Nonconvex Minimax Problems with Coupled linear Constraints
Department of Mathematics, College of Sciences, Shanghai University（上海大学数学系）；Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院）；Newtouch Center for Mathematics of Shanghai University（上海大学数学中心）；State Key Laboratory of Mathematical Sciences, Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院数学科学国家重点实验室）；School of Mathematical Sciences, University of Chinese Academy of Sciences（中国科学院大学数学科学学院）
Consider the minimization of a nonconvex differentiable function over a polyhedron. A popular primal-dual first-order method for this problem is to perform a gradient projection iteration for the augmented Lagrangian function and then update the dual multiplier vector using the constraint residual. However, numerical examples show that this approach can exhibit "oscillation" and may not converge. In this paper, we propose a proximal alternating direction method of multipliers for the multi-block version of this problem. A distinctive feature of this method is the introduction of a "smoothed" (…
Primal-Dual Methods for Nonsmooth Nonconvex Optimization with Orthogonality Constraints
H. Milton Stewart School of Industrial and Systems Engineering, Georgia Institute of Technology（佐治亚理工学院H. Milton Stewart工业与系统工程系）；Department of Systems Engineering and Engineering Management, The Chinese University of Hong Kong（香港中文大学系统工程与工程管理系）
Stochastic gradient descent-ascent (SGDA) is one of the main workhorses for solving finite-sum minimax optimization problems. Most practical implementations of SGDA randomly reshuffle components and sequentially use them (i.e., without-replacement sampling); however, there are few theoretical results on this approach for minimax algorithms, especially outside the easier-to-analyze (strongly-)monotone setups. To narrow this gap, we study the convergence bounds of SGDA with random reshuffling (SGDA-RR) for smooth nonconvex-nonconcave objectives with Polyak-Łojasiewicz (PŁ) geometry. We analyze b…
Adaptive algorithms like AdaGrad and AMSGrad are successful in nonconvex optimization owing to their parameter-agnostic ability -- requiring no a priori knowledge about problem-specific parameters nor tuning of learning rates. However, when it comes to nonconvex minimax optimization, direct extensions of such adaptive optimizers without proper time-scale separation may fail to work in practice. We provide such an example proving that the simple combination of Gradient Descent Ascent (GDA) with adaptive stepsizes can diverge if the primal-dual stepsize ratio is not carefully chosen; hence, a fo…
By applying the perturbation function approach, we propose the Lagrangian and the conjugate duals for minimization problems of the sum of two, generally nonconvex, functions. The main tools are the $Φ$-convexity theory and minimax theorems for $Φ$-convex functions. We provide conditions ensuring zero duality gap and introduce $Φ$-Karush-Kuhn-Tucker conditions that characterize solutions to primal and dual problems. We also discuss the relationship between the dual problems introduced in the present investigation and some conjugate-type duals existing in the literature.
We study nonconvex quadratic problems (QPs) with quadratic separable constraints, where these constraints can be defined both as inequalities or equalities. We derive sufficient conditions for these types of problems to present the S-property, which ultimately guarantees strong duality between the primal and dual problems of the QP. We study the existence of solutions and propose a novel distributed algorithm to solve the problem optimally when the S-property is satisfied. Finally, we illustrate our theoretical results proving that the robust least squares problem with multiple constraints has…
We are to establish necessary conditions (of the primal and dual types) for the set of weak sharp minima of a nonconvex optimization problem on a Riemannian manifold. Here, we are to provide a generalization of some characterizations of weak sharp minima for convex problems on Riemannian manifold introduced by Li et al. (SIAM J. Optim., 21 (2011), pp. 1523--1560) for nonconvex problems. We use the theory of the Fréchet and limiting subdifferentials on Riemannian manifold to give the necessary conditions of the dual type. We also consider a theory of contingent directional derivative and a noti…
The Primal-Dual Hybrid Gradient Method for Semiconvex Splittings
Technical University Munich（慕尼黑工业大学）
本文分析了原始-对偶混合梯度法[Zhu and Chan 2008，Pock，Cremers，Bischof and Chambolle 2009，Esser，Zhang and Chan 2010，Chambolle and Pock 2011]最近的一个重新表述，它允许将其应用于非凸正则化，如[Strekalovskiy and Cremers 2014]中首次提出的截断二次惩罚。特别是，它研究变分问题的能量被最小化，可以写为$G(u) + F(Ku)$，其中$G$是凸，$F$凸，和$K$是一个线性算子。我们研究的方法，并证明收敛的情况下，非凸性的$F$补偿的强凸性的$G$。收敛性证明产生了一个有趣的要求，算法参数的选择，我们不仅是足够的，但必要的。此外，我们证明了迭代在更弱的条件下的有界性。最后，我们证明了算法的有效性和收敛性超出了理论保证在几个数值实验。
In this paper, we consider a proximal linearized alternating direction method of multipliers (PL-ADMM) for solving linearly constrained nonconvex and possibly nonsmooth optimization problems. The algorithm is generalized by using variable metric proximal terms in the primal updates and an over-relaxation stepsize in the multiplier update. We prove that the sequence generated by this method is bounded and its limit points are critical points. Under the powerful Kurdyka-{Łojasiewicz} properties we prove that the sequence has a finite length thus converges, and we drive its convergence rates.
Error bound analysis, which estimates the distance of a point to the solution set of an optimization problem using the optimality residual, is a powerful tool for the analysis of first-order optimization algorithms. In this paper, we use global error bound analysis to study the iteration complexity of a first-order algorithm for a linearly constrained nonconvex minimization problem. we develop a global dual error bound analysis for a regularized version of this nonconvex problem by using a novel ``decomposition'' technique. Equipped with this global dual error bound, we prove that a suitably d…
This paper investigates how to accelerate the convergence of distributed optimization algorithms on nonconvex problems with zeroth-order information available only. We propose a zeroth-order (ZO) distributed primal-dual stochastic coordinates algorithm equipped with "powerball" method to accelerate. We prove that the proposed algorithm has a convergence rate of $\mathcal{O}(\sqrt{p}/\sqrt{nT})$ for general nonconvex cost functions. We consider solving the generation of adversarial examples from black-box DNNs problem to compare with the existing state-of-the-art centralized and distributed ZO …
In this paper, we consider the problem of minimizing the sum of nonconvex and possibly nonsmooth functions over a connected multi-agent network, where the agents have partial knowledge about the global cost function and can only access the zeroth-order information (i.e., the functional values) of their local cost functions. We propose and analyze a distributed primal-dual gradient-free algorithm for this challenging problem. We show that by appropriately choosing the parameters, the proposed algorithm converges to the set of first order stationary solutions with a provable global sublinear con…
Minimax problems, such as generative adversarial network, adversarial training, and fair training, are widely solved by a multi-step gradient descent ascent (MGDA) method in practice. However, its convergence guarantee is limited. In this paper, inspired by the primal-dual hybrid gradient method, we propose a new semi-anchoring (SA) technique for the MGDA method. This makes the MGDA method find a stationary point of a structured nonconvex-nonconcave composite minimax problem; its saddle-subdifferential operator satisfies the weak Minty variational inequality condition. The resulting method, na…
Although originally designed and analyzed for convex problems, the alternating direction method of multipliers (ADMM) and its close relatives, Douglas-Rachford splitting (DRS) and Peaceman-Rachford splitting (PRS), have been observed to perform remarkably well when applied to certain classes of structured nonconvex optimization problems. However, partial global convergence results in the nonconvex setting have only recently emerged. In this paper we show how the Douglas-Rachford envelope (DRE), introduced in 2014, can be employed to unify and considerably simplify the theory for devising globa…
We study a special class of non-convex quadratic programs subject to two (possibly indefinite) quadratic constraints when the level sets of the constraint functions are {\it not} arranged {\it alternatively.} It is shown in the paper that this class of problems admit strong duality following a tight SDP relaxation, without assuming primal or dual Slater conditions. Our results cover Ye and Zhang's development in 2003 and the generalized trust region subproblems (GTRS) as special cases. Through the novel geometric view and some simple examples, we can explain why the problem becomes very hard w…
Developing Lagrangian-based Methods for Nonsmooth Nonconvex Optimization
Institute of Operational Research and Analytics, National University of Singapore（新加坡国立大学运营研究与分析研究所）；School of Computer and Computing Science, Hangzhou City University（杭州城市大学计算机与计算科学学院）
A class of non-smooth and non-convex optimization problems with penalty constraints linked to variational inequalities (VI) is studied with respect to its shape differentiability. The specific problem stemming from quasi-brittle fracture describes an elastic body with a Barenblatt cohesive crack under the inequality condition of non-penetration at the crack faces. Based on the Lagrange approach and using smooth penalization with the Lavrentiev regularization, a formula for the shape derivative is derived. The explicit formula contains both primal and adjoint states and is useful for finding de…
