arXiv 1605.033102016-05-11stat.MEmath.STstat.MLstat.TH

Asymptotic equivalence of regularization methods in thresholded parameter space

Yingying Fan, Jinchi Lv

备注：: 39 pages, 3 figures
发表信息：: Journal of the American Statistical Association 108, 1044-1061
DOI：: 10.1080/01621459.2013.803972

英文摘要：

High-dimensional data analysis has motivated a spectrum of regularization methods for variable selection and sparse modeling, with two popular classes of convex ones and concave ones. A long debate has been on whether one class dominates the other, an important question both in theory and to practitioners. In this paper, we characterize the asymptotic equivalence of regularization methods, with general penalty functions, in a thresholded parameter space under the generalized linear model setting, where the dimensionality can grow up to exponentially with the sample size. To assess their performance, we establish the oracle inequalities, as in Bickel, Ritov and Tsybakov (2009), of the global minimizer for these methods under various prediction and variable selection losses. These results reveal an interesting phase transition phenomenon. For polynomially growing dimensionality, the $L_1$-regularization method of Lasso and concave methods are asymptotically equivalent, having the same convergence rates in the oracle inequalities. For exponentially growing dimensionality, concave methods are asymptotically equivalent but have faster convergence rates than the Lasso. We also establish a stronger property of the oracle risk inequalities of the regularization methods, as well as the sampling properties of computable solutions. Our new theoretical results are illustrated and justified by simulation and real data examples.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

阅读原文

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

本文内容

英文摘要相关论文

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

继续探索

相关论文

High dimensional thresholded regression and shrinkage effect

High-dimensional sparse modeling via regularization provides a powerful tool for analyzing large-scale data sets and obtaining meaningful, interpretable models. The use of nonconvex penalty functions shows advantage in selecting important features in high dimensions, but the global optimality of such methods still demands more understanding. In this paper, we consider sparse regression with hard-thresholding penalty, which we show to give rise to thresholded regression. This approach is motivated by its close connection with the $L_0$-regularization, which can be unrealistic to implement in pr…

2016-05-11stat.MEstat.ML

Thresholded Adaptive Validation: Tuning the Graphical Lasso for Graph Recovery

Ruhr University Bochum（波鸿鲁尔大学）

Many Machine Learning algorithms are formulated as regularized optimization problems, but their performance hinges on a regularization parameter that needs to be calibrated to each application at hand. In this paper, we propose a general calibration scheme for regularized optimization problems and apply it to the graphical lasso, which is a method for Gaussian graphical modeling. The scheme is equipped with theoretical guarantees and motivates a thresholding pipeline that can improve graph recovery. Moreover, requiring at most one line search over the regularization path, the calibration schem…

2021-03-30cs.LGstat.MEstat.ML

Graphical Lasso and Thresholding: Equivalence and Closed-form Solutions

Graphical Lasso (GL) is a popular method for learning the structure of an undirected graphical model, which is based on an $l_1$ regularization technique. The objective of this paper is to compare the computationally-heavy GL technique with a numerically-cheap heuristic method that is based on simply thresholding the sample covariance matrix. To this end, two notions of sign-consistent and inverse-consistent matrices are developed, and then it is shown that the thresholding and GL methods are equivalent if: (i) the thresholded sample covariance matrix is both sign-consistent and inverse-consis…

2019-06-28stat.ML

Regularization methods for learning incomplete matrices

We use convex relaxation techniques to provide a sequence of solutions to the matrix completion problem. Using the nuclear norm as a regularizer, we provide simple and very efficient algorithms for minimizing the reconstruction error subject to a bound on the nuclear norm. Our algorithm iteratively replaces the missing elements with those obtained from a thresholded SVD. With warm starts this allows us to efficiently compute an entire regularization path of solutions.

2009-06-11stat.COstat.ML

用于人类活动序列可解释降维的多项式阈值 LASSO

Multinomial thresholded LASSO for interpretable dimension reduction of human activity sequences

针对高维、含复杂时序依赖的人类活动分类序列，本文将关键位置识别建模为回归并比较多种正则化方法，发现阈值 LASSO 在智能手机时间使用数据上表现更优。

2025-07-23stat.ME

Characterization of the equivalence of robustification and regularization in linear and matrix regression

MIT（麻省理工学院）

The notion of developing statistical methods in machine learning which are robust to adversarial perturbations in the underlying data has been the subject of increasing interest in recent years. A common feature of this work is that the adversarial robustification often corresponds exactly to regularization methods which appear as a loss function plus a penalty. In this paper we deepen and extend the understanding of the connection between robustification and regularization (as achieved by penalization) in regression problems. Specifically, (a) in the context of linear regression, we character…

2017-02-25cs.LGmath.OCmath.ST

更多相关论文收起更多论文

Exact covariance thresholding into connected components for large-scale Graphical Lasso

We consider the sparse inverse covariance regularization problem or graphical lasso with regularization parameter $ρ$. Suppose the co- variance graph formed by thresholding the entries of the sample covariance matrix at $ρ$ is decomposed into connected components. We show that the vertex-partition induced by the thresholded covariance graph is exactly equal to that induced by the estimated concentration graph. This simple rule, when used as a wrapper around existing algorithms, leads to enormous performance gains. For large values of $ρ$, our proposal splits a large graphical lasso problem int…

2011-09-15stat.COstat.ML

A Comparison of Hamming Errors of Representative Variable Selection Methods

Lasso is a celebrated method for variable selection in linear models, but it faces challenges when the variables are moderately or strongly correlated. This motivates alternative approaches such as using a non-convex penalty, adding a ridge regularization, or conducting a post-Lasso thresholding. In this paper, we compare Lasso with 5 other methods: Elastic net, SCAD, forward selection, thresholded Lasso, and forward backward selection. We measure their performances theoretically by the expected Hamming error, assuming that the regression coefficients are iid drawn from a two-point mixture and…

2022-03-28math.STstat.TH

在AMP框架中使用Lasso统计量的经验贝叶斯变量选择

Empirical Bayes Variable Selection with Lasso Statistics in the AMP Framework

本文在AMP框架中提出基于经验贝叶斯的最优变量选择方法，通过估计局部假发现率来改进Lasso统计量的使用，实现显著的功率提升。

2026-02-09math.STstat.TH

IRLS与黏菌：等价性与收敛性

IRLS and Slime Mold: Equivalence and Convergence

École Polytechnique Fédérale de Lausanne (EPFL)（洛桑联邦理工学院）

本文探讨了信号处理中的IRLS算法与生物中的黏菌动力学的等价性，证明了两者为同一高维动力系统的投影，并通过阻尼IRLS算法得出收敛性与复杂度界。

2016-01-12cs.DScs.ETcs.IT

特征向量空间滤波核范数矩阵补全及其在空气质量数据中的应用

Eigenvector Spatial Filters Nuclear Norm Matrix Completion with Application to Air Quality Data

针对环境面板数据中缺失观测的可靠重建问题，提出特征向量空间滤波核范数矩阵补全方法，通过引入Moran型特征向量捕获空间自相关，采用块坐标下降法求解多凸优化问题，在模拟和实际空气质量数据中显著提升插补精度。

2026-08-25stat.APstat.ME

An Equivalence between the Lasso and Support Vector Machines

ETH Zürich（苏黎世联邦理工学院）

We investigate the relation of two fundamental tools in machine learning and signal processing, that is the support vector machine (SVM) for classification, and the Lasso technique used in regression. We show that the resulting optimization problems are equivalent, in the following sense. Given any instance of an $\ell_2$-loss soft-margin (or hard-margin) SVM, we construct a Lasso instance having the same optimal solutions, and vice versa. As a consequence, many existing optimization algorithms for both SVMs and Lasso can also be applied to the respective other problem instances. Also, the equ…

2014-04-25cs.LGstat.ML

Approximate Leave-One-Out for Fast Parameter Tuning in High Dimensions

Columbia University（哥伦比亚大学）；Massachusetts Institute of Technology（麻省理工学院）；Google Research（谷歌研究院）

Consider the following class of learning schemes: $$\hat{\boldsymbolβ} := \arg\min_{\boldsymbolβ}\;\sum_{j=1}^n \ell(\boldsymbol{x}_j^\top\boldsymbolβ; y_j) + λR(\boldsymbolβ),\qquad\qquad (1) $$ where $\boldsymbol{x}_i \in \mathbb{R}^p$ and $y_i \in \mathbb{R}$ denote the $i^{\text{th}}$ feature and response variable respectively. Let $\ell$ and $R$ be the loss function and regularizer, $\boldsymbolβ$ denote the unknown weights, and $λ$ be a regularization parameter. Finding the optimal choice of $λ$ is a challenging problem in high-dimensional regimes where both $n$ and $p$ are large. We pro…

2018-07-07cs.LGstat.ML

通过网络辅助贝叶斯正则化方法联合识别空间可变基因

Joint identification of spatially variable genes via a network-assisted Bayesian regularization approach

本研究针对现有空间可变基因识别方法未兼顾基因间网络结构、易受细胞组成混杂变异影响的问题，提出带阈值化图拉普拉斯正则化的贝叶斯方法，基于零膨胀负二项分布适配数据特性，经模拟和真实数据验证性能优异。

2025-10-03stat.ME