arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1104.4824stat.MLcs.ITmath.IT
收藏

高维统计恢复梯度方法的快速全局收敛性

Fast global convergence of gradient methods for high-dimensional statistical recovery

Alekh Agarwal, Sahand N. Negahban, Martin J. Wainwright

摘要:

许多统计$M$ -估计器是基于凸优化问题,该问题由数据相关损失函数与基于范数的正则化器的组合形成。我们分析了解决这些问题的投影梯度和复合梯度方法的收敛速度,工作在一个高维的框架,允许数据维度$\pdim$增长(并可能超过)的样本大小$\numobs$。这种高维结构排除了通常的全局假设-即强凸性和光滑性条件-这是经典优化分析的基础。我们定义了适当的限制这些条件的版本,并表明,他们满足各种统计模型的高概率。在这些条件下,我们的理论保证,投影梯度下降具有全局几何收敛速度高达模型的\emph{statistical precision},这意味着真正的未知参数$θ^*$和最优解$\hatθ$之间的典型距离。该结果比先前的收敛结果明显更尖锐,先前的收敛结果产生次线性收敛或仅达到噪声水平的线性收敛。我们的分析适用于广泛的$M$ -估计量和统计模型,包括使用Lasso的稀疏线性回归($\ell_1$ -正则化回归);用于块稀疏性的Lasso组;正则化的对数线性模型;使用核范数正则化的低秩矩阵恢复;和矩阵分解。总的来说,我们的分析揭示了高维估计中统计精度和计算效率之间的有趣联系。

英文摘要:

Many statistical $M$-estimators are based on convex optimization problems formed by the combination of a data-dependent loss function with a norm-based regularizer. We analyze the convergence rates of projected gradient and composite gradient methods for solving such problems, working within a high-dimensional framework that allows the data dimension $\pdim$ to grow with (and possibly exceed) the sample size $\numobs$. This high-dimensional structure precludes the usual global assumptions---namely, strong convexity and smoothness conditions---that underlie much of classical optimization analysis. We define appropriately restricted versions of these conditions, and show that they are satisfied with high probability for various statistical models. Under these conditions, our theory guarantees that projected gradient descent has a globally geometric rate of convergence up to the \emph{statistical precision} of the model, meaning the typical distance between the true unknown parameter $θ^*$ and an optimal solution $\hatθ$. This result is substantially sharper than previous convergence results, which yielded sublinear convergence, or linear convergence only up to the noise level. Our analysis applies to a wide range of $M$-estimators and statistical models, including sparse linear regression using Lasso ($\ell_1$-regularized regression); group Lasso for block sparsity; log-linear models with regularization; low-rank matrix recovery using nuclear norm regularization; and matrix decomposition. Overall, our analysis reveals interesting connections between statistical precision and computational efficiency in high-dimensional estimation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑