Many statistical $M$-estimators are based on convex optimization problems formed by the combination of a data-dependent loss function with a norm-based regularizer. We analyze the convergence rates of projected gradient and composite gradient methods for solving such problems, working within a high-dimensional framework that allows the data dimension $\pdim$ to grow with (and possibly exceed) the sample size $\numobs$. This high-dimensional structure precludes the usual global assumptions---namely, strong convexity and smoothness conditions---that underlie much of classical optimization analysis. We define appropriately restricted versions of these conditions, and show that they are satisfied with high probability for various statistical models. Under these conditions, our theory guarantees that projected gradient descent has a globally geometric rate of convergence up to the \emph{statistical precision} of the model, meaning the typical distance between the true unknown parameter $θ^*$ and an optimal solution $\hatθ$. This result is substantially sharper than previous convergence results, which yielded sublinear convergence, or linear convergence only up to the noise level. Our analysis applies to a wide range of $M$-estimators and statistical models, including sparse linear regression using Lasso ($\ell_1$-regularized regression); group Lasso for block sparsity; log-linear models with regularization; low-rank matrix recovery using nuclear norm regularization; and matrix decomposition. Overall, our analysis reveals interesting connections between statistical precision and computational efficiency in high-dimensional estimation.
Accelerating Ill-conditioned Hankel Matrix Recovery via Structured Newton-like Descent
Department of Statistics and Data Science and Department of Computer Science, University of Central Florida(统计与数据科学系和计算机科学系,中央佛罗里达大学);Department of Computational Mathematics, Science and Engineering and Department of Mathematics, Michigan State University(计算数学、科学与工程系和数学系,密歇根州立大学);School of Mathematics and Statistics, and Hubei Key Laboratory of Computational Science, Wuhan University(数学与统计学系,和湖北计算科学重点实验室,武汉大学);School of Artificial Intelligence, Wuhan University, National Center for Applied Mathematics in Hubei, and Hubei Key Laboratory of Computational Science, Wuhan, China(人工智能学院,武汉大学,湖北应用数学中心,和湖北计算科学重点实验室,中国武汉);Institute for Advanced Study, Shenzhen University(先进研究学院,深圳大学)
A refined convergence analysis of pDCA$_e$ with applications to simultaneous sparse recovery and outlier detection
We consider the problem of minimizing a difference-of-convex (DC) function, which can be written as the sum of a smooth convex function with Lipschitz gradient, a proper closed convex function and a continuous possibly nonsmooth concave function . We refine the convergence analysis in [38] for the proximal DC algorithm with extrapolation (pDCA $_e$ ) and show that the whole sequence generated by the algorithm is convergent when the objective is level-bounded, { \em without} imposing differentiability assumptions in the concave part. Our analysis is based on a new potential function and we assu…
Fast Signal Recovery from Saturated Measurements by Linear Loss and Nonconvex Penalties
Institute of Image Processing and Pattern Recognition, Shanghai Jiao Tong University(上海交通大学图像处理与模式识别研究所);University of Electronic Science and Technology of China(电子科技大学);Michigan State University(密歇根州立大学)
Tight Convergence Rates for Online Distributed Linear Estimation with Adversarial Measurements
Dept. of Computer Science and Automation, Indian Institute of Science, Bengaluru(印度科学理工学院计算机科学与自动化系);Department of Computer Science, IMT Atlantique, Plouzané(IMT Atlantique计算机科学系)