Ciamac C. Moallemi, Benjamin Van Roy
We establish that the min-sum message-passing algorithm and its asynchronous variants converge for a large class of unconstrained convex optimization problems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
继续探索
We show that Dykstra's splitting for projecting onto the intersection of convex sets can be extended to minimize the sum of convex functions and a regularizing quadratic. We give conditions for which convergence to the primal minimizer holds so that more than one convex function can be minimized at a time, the convex functions are not necessarily sampled in a cyclic manner, and the SHQP strategy for problems involving the intersection of more than one convex set can be applied. When the sum does not involve the regularizing quadratic, we discuss an approximate proximal point method combined wi…
This paper extends the algorithm schemes proposed in \cite{Nesterov2007a} and \cite{Nesterov2007b} to the minimization of the sum of a composite objective function and a convex function. Two proximal point-type schemes are provided and their global convergence is investigated. The worst case complexity bound is also estimated under certain Lipschitz conditions and nondegeneratedness. The algorithm is then accelerated to get a faster convergence rate for the strongly convex case.
On the Set of Possible Minimizers of a Sum of Convex Functions
该研究针对仅已知各凸函数被加项极小值点位置的场景，利用函数类插值充要条件，精确刻画了凸函数和的可能极小值点集合，还推导了单位球约束下和函数极小值点范数的紧界，证明更简洁直接。
Convex sets approximable as the sum of a compact set and a cone
本文研究凸集在 Hausdorff 距离下近似为紧凸集与闭凸锥 Minkowski 和的问题，通过支撑函数刻画近似 Motzkin 可分解集，并给出其等价的闭定义域与连续性条件。
Convex regularization and subdifferential calculus
该文研究局部凸空间上函数之和的闭凸包正则化问题，给出和的闭凸包可分解的ε-次微分条件及凸函数的精确演算规则，并提出Rockafellar凸积分定理的新证明变体。
Optimal Acceleration for Proximal Minimization of the Sum of Convex and Strongly Convex Functions
本文提出了一种加速方法FDR，改进了之前工作的常数，并证明了$\mathcal{O}(1/N^2)$收敛速率和FDR的主导常数都是最优的。
Convexity and SOS-Convexity of Sum of Separable and Biquadratic Quartic Polynomials and Optimization
研究四次或更高次多元多项式凸性判定难题，引入 SPBQ 形式多项式，分析其凸性与 SOS 凸性联系，构造特殊例子，研究相关优化问题，相比一般 SOS 方法有计算优势，可用于凸多项式回归和流体动力学。
This paper presents a special type of distributed optimization problems, where the summation of agents' local cost functions (i.e., global cost function) is convex, but each individual can be non-convex. Unlike most distributed optimization algorithms by taking the advantages of gradient, the considered problem is allowed to be non-smooth, and the gradient information is unknown to the agents. To solve the problem, a Gaussian-smoothing technique is introduced and a gradient-free method is proposed. We prove that each agent's iterate approximately converges to the optimal solution both with pro…
Sum-Rate Maximization via Convex Optimization Using Subgradient Projections Onto Nonlinear Spectral Radius Constraint Sets
本文提出了一种基于子梯度投影的凸优化方法，用于在非线性谱半径约束集上实现总速率最大化，适用于无细胞网络等实际场景。
Splitting Proximal Point Algorithms for the Sum of Prox-Convex Functions
本文提出两种分裂近端点算法用于求解近似凸函数和的最小化问题，并通过理论分析和实验验证了其收敛性与有效性。
The sum of ratios problem has a variety of important applications in economics and management science, but it is difficult to globally solve this problem. In this paper, we consider the minimization problem of a sum of a number of nondifferentiable quasi-convex component functions over a closed and convex set, which includes the sum of ratios problem as a special case. The sum of quasi-convex component functions is not necessarily to be quasi-convex, and so, this study goes beyond quasi-convex optimization. Exploiting the structure of the sum-minimization problem, we propose a new incremental …
In view of the minimization of a function which is the sum of a differentiable function $f$ and a convex function $g$ we introduce descent methods which can be viewed as produced by inexact auxiliary problem principleor inexact variable metric forward-backward algorithm. Assuming that the global objective function satisfies the Kurdyka-Lojasiewicz inequalitywe prove the convergence of the proposed algorithm weakening assumptions found in previous works.
In this paper we present a variant of the proximal forward-backward splitting iteration for solving nonsmooth optimization problems in Hilbert spaces, when the objective function is the sum of two nondifferentiable convex functions. The proposed iteration, which will be called Proximal Subgradient Splitting Method, extends the classical subgradient iteration for important classes of problems, exploiting the additive structure of the objective function. The weak convergence of the generated sequence was established using different stepsizes and under suitable assumptions. Moreover, we analyze t…
The Minimizer of the Sum of Two Strongly Convex Functions
本文针对分布式优化中无法获取函数完整信息的问题，在梯度范数约束下，为两个强凸函数之和的极小值点所在区域提供了边界与内部完全相同的外部和内部近似。
The flexibility of active ($p$) and reactive power ($q$) consumption in distributed energy resources (DERs) can be represented as a (potentially non-convex) set of points in the $p$-$q$ plane. Modeling of the aggregated flexibility in a heterogeneous ensemble of DERs as a Minkowski sum (M-sum) is computationally intractable even for moderately sized populations. In this article, we propose a scalable method of computing the M-sum of the flexibility domains of a heterogeneous ensemble of DERs, which are allowed to be non-convex, non-compact. In particular, the proposed algorithm computes a guar…
The projector onto the Minkowski sum of closed convex sets is generally not equal to the sum of individual projectors. In this work, we provide a complete answer to the question of characterizing the instances where such an equality holds. Our results unify and extend the case of linear subspaces and Zarantonello's results for projectors onto cones. A detailed analysis in the case of convex combinations is also carried out. We establish the partial sum property for projectors onto convex cones, and we also present various examples as well as a detailed analysis in the univariate case.
In this paper, the problem of computing the projection, and therefore the minimum distance, from a point onto a Minkowski sum of general convex sets is studied. Our approach is based on the minimum norm duality theorem originally stated by Nirenberg and the Nesterov smoothing techniques. It is shown that projection points onto a Minkowski sum of sets can be represented as the sum of points on constituent sets so that, at these points, all of the sets share the same normal vector which is the negative of the dual solution. The proposed NESMINO algorithm improves the theoretical bound on number …
Sparsification of sums with respect to convex cones
本文研究了凸锥的稀疏化函数，提出了稀疏化方法的上界，并探讨了其对锥优化问题的影响。
Generalized polyhedral convex sets, generalized polyhedral convex functions on locally convex Hausdorff topological vector spaces, and the related constructions such as sum of sets, sum of functions, directional derivative, infimal convolution, normal cone, conjugate function, subdifferential, are studied thoroughly in this paper. Among other things, we show how a generalized polyhedral convex set can be characterized via the finiteness of the number of its faces. In addition, it is proved that the infimal convolution of a generalized polyhedral convex function and a polyhedral convex function…
Many problems arising in image processing and signal recovery with multi-regularization can be formulated as minimization of a sum of three convex separable functions. Typically, the objective function involves a smooth function with Lipschitz continuous gradient, a linear composite nonsmooth function and a nonsmooth function. In this paper, we propose a primal-dual fixed-point (PDFP) scheme to solve the above class of problems. The proposed algorithm for three block problems is a fully splitting symmetric scheme, only involving explicit gradient and linear operators without inner iteration, w…
Every convex homogeneous polynomial (or form) is nonnegative. Blekherman has shown that there exist convex forms that are not sums of squares via a nonconstructive argument. We provide an explicit example of a convex form of degree four in 272 variables that is not a sum of squares. The form is related to the Cauchy-Schwarz inequality over the octonions. The proof uses symmetry reduction together with the fact (due to Blekherman) that forms of even degree, that are near-constant on the unit sphere, are convex. Using this same connection, we obtain improved bounds on the approximation quality a…
We characterize the solution of a broad class of convex optimization problems that address the reconstruction of a function from a finite number of linear measurements. The underlying hypothesis is that the solution is decomposable as a finite sum of components, where each component belongs to its own prescribed Banach space; moreover, the problem is regularized by penalizing some composite norm of the solution. We establish general conditions for existence and derive the generic parametric representation of the solution components. These representations fall into three categories depending on…
The Douglas-Rachford algorithm is widely used in sparse signal processing for minimizing a sum of two convex functions. In this paper, we consider the case where one of the functions is weakly convex but the other is strongly convex so that the sum is convex. We provide a condition that ensures the convergence of the same Douglas-Rachford iterations, provided that the strongly convex function is smooth. We also present a modified Douglas-Rachford algorithm that does not impose a smoothness condition for the convex function. We then provide a discussion on the convergence speed of the two types…
We present a distributed solution to optimizing a convex function composed of several non-convex functions. Each non-convex function is privately stored with an agent while the agents communicate with neighbors to form a network. We show that coupled consensus and projected gradient descent algorithm proposed in [1] can optimize convex sum of non-convex functions under an additional assumption on gradient Lipschitzness. We further discuss the applications of this analysis in improving privacy in distributed optimization.
A formula for the sub\-differential of the sum of a series of convex functions defined on a Banach space was provided by X. Y. Zheng in 1998. In this paper, besides a slight extension to locally convex spaces of Zheng's results, we provide a formula for the conjugate of a countable sum of convex functions. Then we use these results for calculating the sub\-differentials and the conjugates in two situations related to entropy minimization, and we study a concrete example met in Statistical Physics.
A convex form of degree larger than one is always nonnegative since it vanishes together with its gradient at the origin. In 2007, Parrilo asked if convex forms are always sums of squares. A few years later, Blekherman answered the question in the negative by showing through volume arguments that for high enough number of variables, there must be convex forms of degree as low as 4 that are not sums of squares. Remarkably, no examples are known to date. In this paper, we show that all convex forms in 4 variables and of degree 4 are sums of squares. We also show that if a conjecture of Blekherma…
Global and local approaches for the minimization of a sum of pointwise minima of convex functions
本文研究凸函数逐点最小和的极小化问题，提出双凸重构与松弛交替极小化（r-AM）局部算法，并构建无额外变量的紧凑big-M混合整数凸规划（MICP）全局方法，通过局部邻域求解证明候选点最优性或获得更优重启点。
The Mean-Variance-Skewness-Kurtosis (MVSK) portfolio optimization model is a quartic nonconvex polynomial minimization problem over a polytope, which can be formulated as a Difference-of-Convex (DC) program. In this manuscript, we investigate four DC programming approaches for solving the MVSK model. First, two DC formulations based on the projective DC decomposition and the Difference-of-Convex-Sums-of-Squares (DC-SOS) decomposition are established, where the second one is novel. Then, DCA is applied to solve these DC formulations. The convergence analysis of DCA for the MVSK model is establi…
We propose a flexible approach for computing the resolvent of the sum of weakly monotone operators in real Hilbert spaces. This relies on splitting methods where strong convergence is guaranteed. We also prove linear convergence under Lipschitz continuity assumption. The approach is then applied to computing the proximity operator of the sum of weakly convex functions, and particularly to finding the best approximation to the intersection of convex sets.
In a Hilbert framework, we introduce continuous and discrete dynamical systems which aim at solving inclusions governed by structured monotone operators $A=\partialΦ+B$, where $\partialΦ$ is the subdifferential of a convex lower semicontinuous function $Φ$, and $B$ is a monotone cocoercive operator. We first consider the extension to this setting of the regularized Newton dynamic with two potentials. Then, we revisit some related dynamical systems, namely the semigroup of contractions generated by $A$, and the continuous gradient projection dynamic. By a Lyapunov analysis, we show the converge…
We consider cooperative multi-agent consensus optimization problems over both static and time-varying communication networks, where only local communications are allowed. The objective is to minimize the sum of agent-specific possibly non-smooth composite convex functions over agent-specific private conic constraint sets; hence, the optimal consensus decision should lie in the intersection of these private sets. Assuming the sum function is strongly convex, we provide convergence rates in suboptimality, infeasibility and consensus violation; examine the effect of underlying network topology on…
We consider the problem of minimizing a sum of clipped convex functions; applications include clipped empirical risk minimization and clipped control. While the problem of minimizing the sum of clipped convex functions is NP-hard, we present some heuristics for approximately solving instances of these problems. These heuristics can be used to find good, if not global, solutions and appear to work well in practice. We also describe an alternative formulation, based on the perspective transformation, which makes the problem amenable to mixed-integer convex programming and yields computationally …
Seriation seeks to reconstruct a linear order between variables using unsorted, pairwise similarity information. It has direct applications in archeology and shotgun gene sequencing for example. We write seriation as an optimization problem by proving the equivalence between the seriation and combinatorial 2-SUM problems on similarity matrices (2-SUM is a quadratic minimization problem over permutations). The seriation problem can be solved exactly by a spectral algorithm in the noiseless case and we derive several convex relaxations for 2-SUM to improve the robustness of seriation solutions i…
CMA Light: a novel Minibatch Algorithm for large-scale non convex finite sum optimization
针对深度神经网络训练中的非凸有限和优化问题，提出CMA Light小批量梯度算法，消除每次迭代的全目标函数评估需求，在温和假设下证明全局收敛，并在实验中展现优于多数先进优化器的计算效率。
The classical Heron problem states: \emph{on a given straight line in the plane, find a point $C$ such that the sum of the distances from $C$ to the given points $A$ and $B$ is minimal}. This problem can be solved using standard geometry or differential calculus. In the light of modern convex analysis, we are able to investigate more general versions of this problem. In this paper we propose and solve the following problem: on a given nonempty closed convex subset of $\R^s$, find a point such that the sum of the distances from that point to $n$ given nonempty closed convex subsets of $\R^s$ is…
In this paper, we consider solving a class of convex optimization problem which minimizes the sum of three convex functions $f(x)+g(x)+h(Bx)$, where $f(x)$ is differentiable with a Lipschitz continuous gradient, $g(x)$ and $h(x)$ have a closed-form expression of their proximity operators and $B$ is a bounded linear operator. This type of optimization problem has wide application in signal recovery and image processing. To make full use of the differentiability function in the optimization problem, we take advantage of two operator splitting methods: the forward-backward splitting method and th…
A popular approach to minimize a finite-sum of convex functions is stochastic gradient descent (SGD) and its variants. Fundamental research questions associated with SGD include: (i) To find a lower bound on the number of times that the gradient oracle of each individual function must be assessed in order to find an $ε$-minimizer of the overall objective; (ii) To design algorithms which guarantee to find an $ε$-minimizer of the overall objective in expectation at no more than a certain number of times (in terms of $1/ε$) that the gradient oracle of each functions needs to be assessed (i.e., up…
We consider potentially non-convex optimization problems, for which optimal rates of approximation depend on the dimension of the parameter space and the smoothness of the function to be optimized. In this paper, we propose an algorithm that achieves close to optimal a priori computational guarantees, while also providing a posteriori certificates of optimality. Our general formulation builds on infinite-dimensional sums-of-squares and Fourier analysis, and is instantiated on the minimization of multivariate periodic functions.
Many convex optimization problems have structured objective function written as a sum of functions with different types of oracles (full gradient, coordinate derivative, stochastic gradient) and different evaluation complexity of these oracles. In the strongly convex case these functions also have different condition numbers, which eventually define the iteration complexity of first-order methods and the number of oracle calls required to achieve given accuracy. Motivated by the desire to call more expensive oracle less number of times, in this paper we consider minimization of a sum of two fu…
In this paper, we introduce a nonlinear optimization problem whose objective function is the convex log-sum-exp function and the feasible region is defined as a system of fuzzy relational inequalities (FRI) defined by the Lukasiewicz t-norm. Some necessary and sufficient conditions are derived to determine the feasibility of the problem. The feasible solution set is characterized in terms of a finite number of closed convex cells. Since the feasible solutions set of FRIs is non-convex, conventional methods may not be directly employed. An algorithm is presented for solving this nonlinear probl…
In this work, using Moreau envelopes, we define a complete metric for the set of proper lower semicontinuous convex functions. Under this metric, the convergence of each sequence of convex functions is epi-convergence. We show that the set of strongly convex functions is dense but it is only of the first category. On the other hand, it is shown that the set of convex functions with strong minima is of the second category.
Many classical algorithms are found until several years later to outlive the confines in which they were conceived, and continue to be relevant in unforeseen settings. In this paper, we show that SVRG is one such method: being originally designed for strongly convex objectives, it is also very robust in non-strongly convex or sum-of-non-convex settings. More precisely, we provide new analysis to improve the state-of-the-art running times in both settings by either applying SVRG or its novel variant. Since non-strongly convex objectives include important examples such as Lasso or logistic regre…
Derivative-free optimization (DFO) has recently gained a lot of momentum in machine learning, spawning interest in the community to design faster methods for problems where gradients are not accessible. While some attention has been given to the concept of acceleration in the DFO literature, existing stochastic algorithms for objective functions with a finite-sum structure have not been shown theoretically to achieve an accelerated rate of convergence. Algorithms that use acceleration in such a setting are prone to instabilities, making it difficult to reach convergence. In this work, we explo…
We address the minimization of the sum of a proper, convex and lower semicontinuous with a (possibly nonconvex) smooth function from the perspective of an implicit dynamical system of forward-backward type. The latter is formulated by means of the gradient of the smooth function and of the proximal point operator of the nonsmooth one. The trajectory generated by the dynamical system is proved to asymptotically converge to a critical point of the objective, provided a regularization of the latter satisfies the Kurdyka-Łojasiewicz property. Convergence rates for the trajectory in terms of the Ło…
The most important open problem in Monotone Operator Theory concerns the maximal monotonicity of the sum of two maximally monotone operators provided that Rockafellar's constraint qualification holds. In this paper, we prove the maximal monotonicity of $A+\partial f$ provided that $A$ is a maximally monotone linear relation, and $f$ is a proper lower semicontinuous convex function satisfying $\dom A\cap\inte\dom \partial f\neq\varnothing$. Moreover, $A+\partial f$ is of type (FPV). The maximal monotonicity of $A+\partial f$ when $\intdom A\cap\dom \partial f\neq\varnothing$ follows from a resu…
Convergence Analysis of the ProbAbilistic Gradient Estimator Algorithm for Weakly Convex Finite-Sum Optimization
研究概率梯度估计器算法（PAGE）在τ-弱凸函数框架下，给出非凸和平凸情况间连续插值，建立新收敛率，表明随τ减小复杂度改善。
Convex Ternary Quartics Are SOS-Convex
本文证明了三元四次凸型是平方和凸的，这是希尔伯特1888年定理的凸类似，并指出利用Hessian矩阵结构对证明至关重要。
Monotone Near-Zero-Sum Games: A Generalization of Convex-Concave Minimax
本文提出单调近零和游戏模型，通过将近零和游戏转化为零和子问题，改进了梯度复杂性，并展示了其在实际游戏场景中的应用。
We propose an adaptive variance-reduction method, called AdaSpider, for minimization of $L$-smooth, non-convex functions with a finite-sum structure. In essence, AdaSpider combines an AdaGrad-inspired [Duchi et al., 2011, McMahan & Streeter, 2010], but a fairly distinct, adaptive step-size schedule with the recursive stochastic path integrated estimator proposed in [Fang et al., 2018]. To our knowledge, Adaspider is the first parameter-free non-convex variance-reduction method in the sense that it does not require the knowledge of problem-dependent parameters, such as smoothness constant $L$, …
Structured problems arise in many applications. To solve these problems, it is important to leverage the structure information. This paper focuses on convex problems with a finite-sum compositional structure. Finite-sum problems appear as the sample average approximation of a stochastic optimization problem and also arise in machine learning with a huge amount of training data. One popularly used numerical approach for finite-sum problems is the stochastic gradient method (SGM). However, the additional compositional structure prohibits easy access to unbiased stochastic approximation of the gr…
We develop a class of algorithms, as variants of the stochastically controlled stochastic gradient (SCSG) methods (Lei and Jordan, 2016), for the smooth non-convex finite-sum optimization problem. Assuming the smoothness of each component, the complexity of SCSG to reach a stationary point with $\mathbb{E} \|\nabla f(x)\|^{2}\le ε$ is $O\left (\min\{ε^{-5/3}, ε^{-1}n^{2/3}\}\right)$, which strictly outperforms the stochastic gradient descent. Moreover, SCSG is never worse than the state-of-the-art methods based on variance reduction and it significantly outperforms them when the target accurac…
On the Location of the Minimizer of the Sum of Two Strongly Convex Functions
本文研究了两个强凸函数之和的极小值点位置与其各函数性质之间的关系，提出了一个上界，以确定极小值点所在的区域，并考虑了两种不同梯度约束场景下的极小值点边界特性。
This paper studies the application of the blended dynamics approach towards distributed optimization problem where the global cost function is given by a sum of local cost functions. The benefits include (i) individual cost function need not be convex as long as the global cost function is strongly convex and (ii) the convergence rate of the distributed algorithm is arbitrarily close to the convergence rate of the centralized one. Two particular continuous-time algorithms are presented using the proportional-integral-type couplings. One has benefit of `initialization-free,' so that agents can …
Finite-sum optimization plays an important role in the area of machine learning, and hence has triggered a surge of interest in recent years. To address this optimization problem, various randomized incremental gradient methods have been proposed with guaranteed upper and lower complexity bounds for their convergence. Nonetheless, these lower bounds rely on certain conditions: deterministic optimization algorithm, or fixed probability distribution for the selection of component functions. Meanwhile, some lower bounds even do not match the upper bounds of the best known methods in certain cases…
The distributed convex optimization problem over the multi-agent system is considered in this paper, and it is assumed that each agent possesses its own cost function and communicates with its neighbours over a sequence of time-varying directed graphs. However, due to some reasons there exist communication delays while agents receive information from other agents, and we are going to seek the optimal value of the sum of agents' loss functions in this case. We desire to handle this problem with the push-sum distributed dual averaging (PS-DDA) algorithm. It is proved that this algorithm converge…
This paper considers stochastic first-order algorithms for convex-concave minimax problems of the form $\min_{\bf x}\max_{\bf y}f(\bf x, \bf y)$, where $f$ can be presented by the average of $n$ individual components which are $L$-average smooth. For $μ_x$-strongly-convex-$μ_y$-strongly-concave setting, we propose a new method which could find a $\varepsilon$-saddle point of the problem in $\tilde{\mathcal O} \big(\sqrt{n(\sqrt{n}+κ_x)(\sqrt{n}+κ_y)}\log(1/\varepsilon)\big)$ stochastic first-order complexity, where $κ_x\triangleq L/μ_x$ and $κ_y\triangleq L/μ_y$. This upper bound is near optim…
In this paper, we introduce a simplified and unified method for finite-sum convex optimization, named \emph{Variance Reduction via Accelerated Dual Averaging (VRADA)}. In both general convex and strongly convex settings, VRADA can attain an $O\big(\frac{1}{n}\big)$-accurate solution in $O(n\log\log n)$ number of stochastic gradient evaluations which improves the best-known result $O(n\log n)$, where $n$ is the number of samples. Meanwhile, VRADA matches the lower bound of the general convex setting up to a $\log\log n$ factor and matches the lower bounds in both regimes $n\le Θ(κ)$ and $n\gg κ…
Zero-Gradient-Sum Algorithms for Distributed Convex Optimization: The Continuous-Time Case
针对固定拓扑无向网络上的无约束可分凸优化问题，提出基于李雅普诺夫函数的连续时间分布式零梯度求和算法，保证渐近收敛到全局最优解，并分析了指数收敛速率与网络拓扑、问题特性及算法参数的关系。
A Near-Optimal Single-Loop Stochastic Algorithm for Convex Finite-Sum Coupled Compositional Optimization
本文提出单循环原始-对偶块坐标随机算法ALEXR，用于凸有限和耦合组合优化，在光滑与非光滑情形下取得改进的收敛速率，并通过复杂度下界证明其近最优性，在GDRO和pAUC任务上验证了效果。
In this work, we study the problem of minimizing the sum of strongly convex functions split over a network of $n$ nodes. We propose the decentralized and asynchronous algorithm ADFS to tackle the case when local functions are themselves finite sums with $m$ components. ADFS converges linearly when local functions are smooth, and matches the rates of the best known finite sum algorithms when executed on a single machine. On several machines, ADFS enjoys a $O (\sqrt{n})$ or $O(n)$ speed-up depending on the leading complexity term as long as the diameter of the network is not too big with respect…
扫码添加微信好友，提出您的宝贵建议 👇
💡 备注请填写：网站反馈
发表机构由 AI 从论文文本抽取，如发现错误，欢迎反馈修正建议。