稀疏线性模型的最优预测坐标可分M-估计量的下界
Optimal prediction for sparse linear models? Lower bounds for coordinate-separable M-estimators
摘要：
对于高维稀疏线性回归问题，已知基于$\ell_0$的估计器可以在没有设计矩阵上的任何条件的情况下实现预测误差的$1/n$“快”速率，而在没有设计矩阵上的限制性条件的情况下，流行的多项式时间方法仅保证$1/\sqrt{n}$“慢”速率。在本文中，我们证明了慢速率是一个广泛的M-估计类的内在。特别地，对于基于最小化最小二乘费用函数以及（可能非凸的）坐标方向可分离正则化器的估计器，总是存在“坏的”局部最优值，使得相关联的预测误差由$1/\sqrt{n}$的恒定倍数下界。对于凸正则化子，这个下界适用于所有全局最优解。该理论适用于许多流行的估计，包括凸$\ell_1$为基础的方法，以及M-估计的基础上非凸正则化，包括SCAD惩罚或MCP正则化。此外，对于广泛的一类非凸正则化，我们表明，坏的局部最优解是非常常见的，在广泛的一类随机初始化的局部最小化算法通常会收敛到一个坏的解决方案。
英文摘要：
For the problem of high-dimensional sparse linear regression, it is known that an $\ell_0$-based estimator can achieve a $1/n$ "fast" rate on the prediction error without any conditions on the design matrix, whereas in absence of restrictive conditions on the design matrix, popular polynomial-time methods only guarantee the $1/\sqrt{n}$ "slow" rate. In this paper, we show that the slow rate is intrinsic to a broad class of M-estimators. In particular, for estimators based on minimizing a least-squares cost function together with a (possibly non-convex) coordinate-wise separable regularizer, there is always a "bad" local optimum such that the associated prediction error is lower bounded by a constant multiple of $1/\sqrt{n}$. For convex regularizers, this lower bound applies to all global optima. The theory is applicable to many popular estimators, including convex $\ell_1$-based methods as well as M-estimators based on nonconvex regularizers, including the SCAD penalty or the MCP regularizer. In addition, for a broad class of nonconvex regularizers, we show that the bad local optima are very common, in that a broad class of local minimization algorithms with random initialization will typically converge to a bad solution.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
