The performance of stochastic gradient descent (SGD), which is the simplest first-order optimizer for training deep neural networks, depends on not only the learning rate but also the batch size. They both affect the number of iterations and the stochastic first-order oracle (SFO) complexity needed for training. In particular, the previous numerical results indicated that, for SGD using a constant learning rate, the number of iterations needed for training decreases when the batch size increases, and the SFO complexity needed for training is minimized at a critical batch size and that it increases once the batch size exceeds that size. Here, we study the relationship between batch size and the iteration and SFO complexities needed for nonconvex optimization in deep learning with SGD using constant or decaying learning rates and show that SGD using the critical batch size minimizes the SFO complexity. We also provide numerical comparisons of SGD with the existing first-order optimizers and show the usefulness of SGD using a critical batch size. Moreover, we show that measured critical batch sizes are close to the sizes estimated from our theoretical results.
University of California, Davis（加州大学戴维斯分校）；University of Waterloo（滑铁卢大学）
We analyze stochastic conditional gradient methods for constrained optimization problems arising in over-parametrized machine learning. We show that one could leverage the interpolation-like conditions satisfied by such models to obtain improved oracle complexities. Specifically, when the objective function is convex, we show that the conditional gradient method requires $\mathcal{O}(ε^{-2})$ calls to the stochastic gradient oracle to find an $ε$-optimal solution. Furthermore, by including a gradient sliding step, we show that the number of calls reduces to $\mathcal{O}(ε^{-1.5})$.
University of California, Davis（加州大学戴维斯分校）；University of Southern California（南加州大学）
In this paper, we study zeroth-order algorithms for minimax optimization problems that are nonconvex in one variable and strongly-concave in the other variable. Such minimax optimization problems have attracted significant attention lately due to their applications in modern machine learning tasks. We first consider a deterministic version of the problem. We design and analyze the Zeroth-Order Gradient Descent Ascent (\texttt{ZO-GDA}) algorithm, and provide improved results compared to existing works, in terms of oracle complexity. We also propose the Zeroth-Order Gradient Descent Multi-Step A…
Unbiased and Biased Variance-Reduced Forward-Reflected-Backward Splitting Methods for Stochastic Composite Inclusions
Department of Statistics and Operations Research（统计与运筹学系）；The University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；Hanes Hall, UNC-Chapel Hill, NC 27599-3260（UNC-教堂山分校318号哈尼斯楼，NC 27599-3260）
Massachusetts Institute of Technology（麻省理工学院）；Purdue University（普渡大学）
Several recent empirical studies demonstrate that important machine learning tasks, e.g., training deep neural networks, exhibit low-rank structure, where the loss function varies significantly in only a few directions of the input space. In this paper, we leverage such low-rank structure to reduce the high computational cost of canonical gradient-based methods such as gradient descent (GD). Our proposed \emph{Low-Rank Gradient Descent} (LRGD) algorithm finds an $ε$-approximate stationary point of a $p$-dimensional function by first identifying $r \leq p$ significant directions, and then estim…
Quantum Algorithms for Bandits with Knapsacks with Improved Regret and Time Complexities
Center on Frontiers of Computing Studies, Peking University School of Computer Science, Peking University（前沿计算研究所以及北京大学计算机科学学院，北京大学）；Guanghua School of Management, Peking University（光华管理学院，北京大学）；Department of Management Science and Engineering, Stanford University（管理科学与工程系，斯坦福大学）
Matching Multi-Loop Complexities with a Single Loop: Optimal Optimization Stationarity and Best-Known Game Stationarity in Nonconvex--Concave Minimax Optimization
Department of Computational Applied Mathematics；Operations Research, Rice University, Houston, TX, USA；Department of Statistics, Texas A\&M University, College Station, TX, USA
Escaping Saddle-Points Faster under Interpolation-like Conditions
Department of Statistics, University of California, Davis（加州大学戴维斯分校统计系）；Department of Management Sciences, University of Waterloo（滑铁卢大学管理科学系）；Department of Computer Science, University of California, Davis（加州大学戴维斯分校计算机科学系）；University of California, Davis（加州大学戴维斯分校）；University of Waterloo（滑铁卢大学）
Accelerated Gradient Tracking over Time-varying Graphs for Decentralized Optimization
Institute of Robotics and Automatic Information Systems, College of Artificial Intelligence, Nankai University（南开大学人工智能学院机器人与自动信息系统研究所）；State Key Lab of General AI, School of Intelligence Science and Technology, Peking University（北京大学智能科学与技术学院通用人工智能省部共建国家重点实验室）
Nanjing University（南京大学）；the University of Iowa（爱荷华大学）；Texas A&M University（得克萨斯农工大学）
Variance reduction techniques such as SPIDER/SARAH/STORM have been extensively studied to improve the convergence rates of stochastic non-convex optimization, which usually maintain and update a sequence of estimators for a single function across iterations. What if we need to track multiple functional mappings across iterations but only with access to stochastic samples of $\mathcal{O}(1)$ functional mappings at each iteration? There is an important application in solving an emerging family of coupled compositional optimization problems in the form of $\sum_{i=1}^m f_i(g_i(\mathbf{w}))$, wher…
The University of Iowa（爱荷华大学）；Nanjing University（南京大学）
In this paper, we consider non-convex stochastic bilevel optimization (SBO) problems that have many applications in machine learning. Although numerous studies have proposed stochastic algorithms for solving these problems, they are limited in two perspectives: (i) their sample complexities are high, which do not match the state-of-the-art result for non-convex stochastic optimization; (ii) their algorithms are tailored to problems with only one lower-level problem. When there are many lower-level problems, it could be prohibitive to process all these lower-level problems at each iteration. To…
Projected gradient methods for nonconvex and stochastic smooth optimization: new complexities and auto-conditioned stepsizes
School of Industrial and Systems Engineering, Georgia Institute of Technology（工业与系统工程学院，佐治亚理工学院）；Department of Mathematical Sciences, Rensselaer Polytechnic Institute（数学科学系，伦塞拉尔理工学院）
We consider stochastic zeroth-order optimization over Riemannian submanifolds embedded in Euclidean space, where the task is to solve Riemannian optimization problem with only noisy objective function evaluations. Towards this, our main contribution is to propose estimators of the Riemannian gradient and Hessian from noisy objective function evaluations, based on a Riemannian version of the Gaussian smoothing technique. The proposed estimators overcome the difficulty of the non-linearity of the manifold constraint and the issues that arise in using Euclidean Gaussian smoothing techniques when …
A Trust-Region Interior-Point Stochastic Sequential Quadratic Programming Method
Department of Mathematics, University of California, Berkeley（加州大学伯克利分校数学系）；Department of IEOR, University of California, Berkeley（加州大学伯克利分校工业工程与运营研究系）；School of Industrial and Systems Engineering, Georgia Tech（佐治亚理工学院工业与系统工程学院）；Department of EECS, University of California, Berkeley（加州大学伯克利分校电子工程与计算机科学系）
