首页/论文/1212.2002

arXiv 1212.20022012-12-20cs.LGmath.OCstat.ML

A simpler approach to obtaining an O(1/t) convergence rate for the projected stochastic subgradient method

INRIA（法国国家信息与自动化研究所）；Département d’Informatique de l’Ecole Normale Supérieure（巴黎高等师范学院计算机系）

Simon Lacoste-Julien, Mark Schmidt, Francis Bach

备注：: 8 pages, 6 figures. Changes with previous version: Added reference to concurrently submitted work arXiv:1212.1824v1; clarifications added; typos corrected; title changed to 'subgradient method' as 'subgradient descent' is misnomer

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

英文摘要相关论文

英文摘要：

In this note, we present a new averaging technique for the projected stochastic subgradient method. By using a weighted average with a weight of t+1 for each iterate w_t at iteration t, we obtain the convergence rate of O(1/t) with both an easy proof and an easy implementation. The new scheme is compared empirically to existing techniques, with similar performance behavior.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

阅读原文

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

本文内容

英文摘要相关论文

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

继续探索

相关论文

On Distributed Non-convex Optimization: Projected Subgradient Method For Weakly Convex Problems in Networks

Texas A&M University（德克萨斯农工大学）

The stochastic subgradient method is a widely-used algorithm for solving large-scale optimization problems arising in machine learning. Often these problems are neither smooth nor convex. Recently, Davis et al. [1-2] characterized the convergence of the stochastic subgradient method for the weakly convex case, which encompasses many important applications (e.g., robust phase retrieval, blind deconvolution, biconvex compressive sensing, and dictionary learning). In practice, distributed implementations of the projected stochastic subgradient method (stoDPSM) are used to speed-up risk minimizati…

2021-02-23cs.LGmath.OCstat.ML

Proximally Guided Stochastic Subgradient Method for Nonsmooth, Nonconvex Problems

Cornell University（康奈尔大学）

In this paper, we introduce a stochastic projected subgradient method for weakly convex (i.e., uniformly prox-regular) nonsmooth, nonconvex functions---a wide class of functions which includes the additive and convex composite classes. At a high-level, the method is an inexact proximal point iteration in which the strongly convex proximal subproblems are quickly solved with a specialized stochastic projected subgradient method. The primary contribution of this paper is a simple proof that the proposed algorithm converges at the same rate as the stochastic gradient method for smooth nonconvex p…

2018-09-18cs.LGmath.OC

Convergence Rates for Deterministic and Stochastic Subgradient Methods Without Lipschitz Continuity

Cornell University（康奈尔大学）；Simons Institute for the Theory of Computing（西蒙斯理论计算研究所）；DIMACS/Simons Collaboration on Bridging Continuous and Discrete Optimization（DIMACS/西蒙斯连续与离散优化桥接合作项目）

We extend the classic convergence rate theory for subgradient methods to apply to non-Lipschitz functions. For the deterministic projected subgradient method, we present a global $O(1/\sqrt{T})$ convergence rate for any convex function which is locally Lipschitz around its minimizers. This approach is based on Shor's classic subgradient analysis and implies generalizations of the standard convergence rates for gradient descent on functions with Lipschitz or Hölder continuous gradients. Further, we show a $O(1/\sqrt{T})$ convergence rate for the stochastic projected subgradient method on convex…

2018-02-26cs.LGmath.OC

Projected Subgradient Methods for Learning Sparse Gaussians

Stanford University（斯坦福大学）

Gaussian Markov random fields (GMRFs) are useful in a broad range of applications. In this paper we tackle the problem of learning a sparse GMRF in a high-dimensional space. Our approach uses the l1-norm as a regularization on the inverse covariance matrix. We utilize a novel projected gradient method, which is faster than previous methods in practice and equal to the best performing of these in asymptotic complexity. We also extend the l1-regularized objective to the problem of sparsifying entire blocks within the inverse covariance matrix. Our methods generalize fairly easily to this case, w…

2012-06-13cs.LGstat.ML

The adaptive projected subgradient method constrained by families of quasi-nonexpansive mappings and its application to online learning

University of Peloponnese（伯罗奔尼撒大学）；Tokyo Institute of Technology（东京工业大学）

Many online, i.e., time-adaptive, inverse problems in signal processing and machine learning fall under the wide umbrella of the asymptotic minimization of a sequence of non-negative, convex, and continuous functions. To incorporate a-priori knowledge into the design, the asymptotic minimization task is usually constrained on a fixed closed convex set, which is dictated by the available a-priori information. To increase versatility towards the usage of the available information, the present manuscript extends the Adaptive Projected Subgradient Method (APSM) by introducing an algorithmic scheme…

2011-08-05cs.ITcs.LGmath.IT

Stochastic model-based minimization of weakly convex functions

Cornell University（康奈尔大学）；University of Washington（华盛顿大学）

We consider a family of algorithms that successively sample and minimize simple stochastic models of the objective function. We show that under reasonable conditions on approximation quality and regularity of the models, any such algorithm drives a natural stationarity measure to zero at the rate $O(k^{-1/4})$. As a consequence, we obtain the first complexity guarantees for the stochastic proximal point, proximal subgradient, and regularized Gauss-Newton methods for minimizing compositions of convex functions with smooth maps. The guiding principle, underlying the complexity guarantees, is tha…

2018-08-26cs.LGmath.OC

更多相关论文收起更多论文

The Strength of Nesterov's Extrapolation in the Individual Convergence of Nonsmooth Optimization

Command and Control Engineering College, Army Engineering University of PLA（中国人民解放军陆军工程大学指挥与控制工程学院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Army Academy Of Artillery and Air Defense（陆军防空兵学院）

The extrapolation strategy raised by Nesterov, which can accelerate the convergence rate of gradient descent methods by orders of magnitude when dealing with smooth convex objective, has led to tremendous success in training machine learning tasks. In this article, the convergence of individual iterates of projected subgradient (PSG) methods for nonsmooth convex optimization problems is theoretically studied based on Nesterov's extrapolation, which we name individual convergence. We prove that Nesterov's extrapolation has the strength to make the individual convergence of PSG optimal for nonsm…

2020-06-08cs.LGstat.ML

Detection of Insider Attacks in Distributed Projected Subgradient Algorithms

Shenzhen University（深圳大学）

The gossip-based distributed algorithms are widely used to solve decentralized optimization problems in various multi-agent applications, while they are generally vulnerable to data injection attacks by internal malicious agents as each agent locally estimates its decent direction without an authorized supervision. In this work, we explore the application of artificial intelligence (AI) technologies to detect internal attacks. We show that a general neural network is particularly suitable for detecting and localizing the malicious agents, as they can effectively explore nonlinear relationship …

2021-01-18cs.ITcs.LGmath.IT

Implicit Bias of Projected Subgradient Method Gives Provable Robust Recovery of Subspaces of Unknown Codimension

Mathematical Institute for Data Science（数据科学数学研究所）；Johns Hopkins University（约翰斯·霍普金斯大学）

Robust subspace recovery (RSR) is a fundamental problem in robust representation learning. Here we focus on a recently proposed RSR method termed Dual Principal Component Pursuit (DPCP) approach, which aims to recover a basis of the orthogonal complement of the subspace and is amenable to handling subspaces of high relative dimension. Prior work has shown that DPCP can provably recover the correct subspace in the presence of outliers, as long as the true dimension of the subspace is known. We show that DPCP can provably solve RSR problems in the {\it unknown} subspace dimension regime, as long…

2022-01-22cs.CVcs.LG

阈值化局部超流扩散

Thresholded Local Hyper-Flow Diffusion

KTH Royal Institute of Technology（瑞典皇家理工学院）；Johns Hopkins University（约翰霍普金斯大学）

提出TL-HFD算法，通过局部活动区域和阈值化边界激活实现超图种子聚类的局部扩散，保证与全局更新等价并给出有限时间对偶次优性界。

2026-06-08cs.LG

A Linearly Convergent Conditional Gradient Algorithm with Applications to Online and Stochastic Optimization

Technion - Israel Inst. of Tech.（以色列理工学院）

Linear optimization is many times algorithmically simpler than non-linear convex optimization. Linear optimization over matroid polytopes, matching polytopes and path polytopes are example of problems for which we have simple and efficient combinatorial algorithms, but whose non-linear convex counterpart is harder and admits significantly less efficient algorithms. This motivates the computational model of convex optimization, including the offline, online and stochastic settings, using a linear optimization oracle. In this computational model we give several new results that improve over the …

2015-08-14cs.LGmath.OCstat.ML

One Sample Stochastic Frank-Wolfe

Yale University（耶鲁大学）；University of Pennsylvania（宾夕法尼亚大学）；The University of Texas at Austin（得克萨斯大学奥斯汀分校）

One of the beauties of the projected gradient descent method lies in its rather simple mechanism and yet stable behavior with inexact, stochastic gradients, which has led to its wide-spread use in many machine learning applications. However, once we replace the projection operator with a simpler linear program, as is done in the Frank-Wolfe method, both simplicity and stability take a serious hit. The aim of this paper is to bring them back without sacrificing the efficiency. In this paper, we propose the first one-sample stochastic Frank-Wolfe algorithm, called 1-SFW, that avoids the need to …

2019-10-10cs.LGmath.OCstat.ML

Stochastic algorithms with geometric step decay converge linearly on sharp functions

Cornell University（康奈尔大学）；Microsoft Research（微软研究院）

Stochastic (sub)gradient methods require step size schedule tuning to perform well in practice. Classical tuning strategies decay the step size polynomially and lead to optimal sublinear rates on (strongly) convex problems. An alternative schedule, popular in nonconvex optimization, is called \emph{geometric step decay} and proceeds by halving the step size after every few epochs. In recent work, geometric step decay was shown to improve exponentially upon classical sublinear rates for the class of \emph{sharp} convex functions. In this work, we ask whether geometric step decay similarly impro…

2019-07-22cs.LGmath.OCstat.ML

约束非凸优化中依赖数据下的一阶方法的收敛性

Convergence of First-Order Methods for Constrained Nonconvex Optimization with Dependent Data

Wisconsin Institute for Discovery（威斯康星发现研究所）；University of Wisconsin–Madison（威斯康星大学麦迪逊分校）

针对依赖数据下的约束光滑非凸优化，提出统一分析框架，将随机投影梯度方法的复杂度从 $\tilde{O}(\varepsilon^{-8})$ 改进至 $\tilde{O}(\varepsilon^{-4})$，并推广至 AdaGrad 等算法，首次实现依赖数据的在线非负矩阵分解。

2023-06-23cs.LGmath.OCstat.ML