In this paper, we revisit the large-scale constrained linear regression problem and propose faster methods based on some recent developments in sketching and optimization. Our algorithms combine (accelerated) mini-batch SGD with a new method called two-step preconditioning to achieve an approximate solution with a time complexity lower than that of the state-of-the-art techniques for the low precision case. Our idea can also be extended to the high precision case, which gives an alternative implementation to the Iterative Hessian Sketch (IHS) method with significantly improved time complexity. Experiments on benchmark and synthetic datasets suggest that our methods indeed outperform existing ones considerably in both the low and high precision cases.
Preconditioned gradient methods are among the most general and powerful tools in optimization. However, preconditioning requires storing and manipulating prohibitively large matrices. We describe and analyze a new structure-aware preconditioning algorithm, called Shampoo, for stochastic optimization over tensor spaces. Shampoo maintains a set of preconditioning matrices, each of which operates on a single dimension, contracting over the remaining dimensions. We establish convergence guarantees in the stochastic convex setting, the proof of which builds upon matrix trace inequalities. Our exper…
Constrained Stochastic Spectral Preconditioning Converges for Nonconvex Objectives
KU Leuven, Department of Electrical Engineering, STADIUS Center for Dynamical Systems, Signal Processing, and Data Analytics（KU莱顿大学，电气工程系，动态系统、信号处理和数据分析中心）；École Polytechnique Fédérale de Lausanne (EPFL), Laboratory for Information and Inference Systems (LIONS)（洛桑联邦理工学院（EPFL），信息与推断系统实验室）；CVN, CentraleSupélec, Université Paris-Saclay, Inria（CVN，中央超导学院，巴黎-萨克雷大学，法国国家科学研究中心）
Faster Eigenvector Computation via Shift-and-Invert Preconditioning
Toyota Technological Institute at Chicago（芝加哥丰田技术研究院）；Princeton University（普林斯顿大学）；UC Berkeley（加州大学伯克利分校）；University of Washington（华盛顿大学）；MIT（麻省理工学院）；Microsoft Research, New England（新英格兰微软研究院）
University of Tokyo（东京大学）；RIKEN Center for Advanced Intelligence Project（理化学研究所先进智能项目中心）；Columbia University（哥伦比亚大学）；RIKEN Center for Brain Science（理化学研究所脑科学中心）；University of Toronto（多伦多大学）；Vector Institute for Artificial Intelligence（向量人工智能研究所）；Google Research（谷歌研究院）
While second order optimizers such as natural gradient descent (NGD) often speed up optimization, their effect on generalization has been called into question. This work presents a more nuanced view on how the \textit{implicit bias} of first- and second-order methods affects the comparison of generalization properties. We provide an exact asymptotic bias-variance decomposition of the generalization error of overparameterized ridgeless regression under a general class of preconditioner $\boldsymbol{P}$, and consider the inverse population Fisher information matrix (used in NGD) as a particular …
We study fast algorithms for statistical regression problems under the strong contamination model, where the goal is to approximately optimize a generalized linear model (GLM) given adversarially corrupted samples. Prior works in this line of research were based on the robust gradient descent framework of Prasad et. al., a first-order method using biased gradient queries, or the Sever framework of Diakonikolas et. al., an iterative outlier-removal method calling a stationary point finder. We present nearly-linear time algorithms for robust regression problems with improved runtime or estimatio…
State University of New York at Buffalo（纽约州立大学布法罗分校）
In this paper we study the differentially private Empirical Risk Minimization (ERM) problem in different settings. For smooth (strongly) convex loss function with or without (non)-smooth regularization, we give algorithms that achieve either optimal or near optimal utility bounds with less gradient complexity compared with previous work. For ERM with smooth convex loss function in high-dimensional ($p\gg n$) setting, we give an algorithm which achieves the upper bound with less gradient complexity than previous ones. At last, we generalize the expected excess empirical risk from convex loss fu…
Efficient Over-parameterized Matrix Sensing from Noisy Measurements via Alternating Preconditioned Gradient Descent
State Key Laboratory of Robotics, Shenyang Institute of Automation, Chinese Academy of Sciences（机器人国家重点实验室，沈阳自动化研究所，中国科学院）；University of Chinese Academy of Sciences（中国科学院大学）；State University of New York at Buffalo（纽约州立大学布法罗分校）；School of Management, Xi’an Jiaotong University（西安交通大学管理学院）
Department of XXX, University of YYY, Location, Country（XXX系，YYY大学，地点，国家）；School of ZZZ, Institute of WWW, Location, Country（ZZZ学院，WWW研究所，地点，国家）；Department of Mathematics, Imperial College London, United Kingdom（数学系，伦敦帝国学院，英国）
