We study the asymptotic properties of the SCAD-penalized least squares estimator in sparse, high-dimensional, linear regression models when the number of covariates may increase with the sample size. We are particularly interested in the use of this estimator for simultaneous variable selection and estimation. We show that under appropriate conditions, the SCAD-penalized least squares estimator is consistent for variable selection and that the estimators of nonzero coefficients have the same asymptotic distribution as they would have if the zero coefficients were known in advance. Simulation studies indicate that this estimator performs well in terms of variable selection and estimation.
This paper studies the non-asymptotic merits of the double $\ell_1$-regularized for heterogeneous overdispersed count data via negative binomial regressions. Under the restricted eigenvalue conditions, we prove the oracle inequalities for Lasso estimators of two partial regression coefficients for the first time, using concentration inequalities of empirical processes. Furthermore, derived from the oracle inequalities, the consistency and convergence rate for the estimators are the theoretical guarantees for further statistical inference. Finally, both simulations and a real data analysis demo…
By the asymptotic oracle property, non-convex penalties represented by minimax concave penalty (MCP) and smoothly clipped absolute deviation (SCAD) have attracted much attentions in high-dimensional data analysis, and have been widely used in signal processing, image restoration, matrix estimation, etc. However, in view of their non-convex and non-smooth characteristics, they are computationally challenging. Almost all existing algorithms converge locally, and the proper selection of initial values is crucial. Therefore, in actual operation, they often combine a warm-starting technique to meet…
We derive asymptotic properties of penalized estimators for singular models for which identifiability may break and the true parameter values can lie on the boundary of the parameter space. Selection consistency of the estimators is also validated. The problem that the true values lie on the boundary is dealt with by our previous results that are applicable to singular models, besides, penalized estimation and non-ergodic statistics. In order to overcome non-identifiability, we consider a suitable penalty such as the non-convex Bridge and the adaptive Lasso that stabilizes the asymptotic behav…
The paper focuses on the automatic selection of the grouped explanatory variables in an high-dimensional model, when the model errors are asymmetric. After introducing the model and notations, we define the adaptive group LASSO expectile estimator for which we prove the oracle properties: the sparsity and the asymptotic normality. Afterwards, the results are generalized by considering the asymmetric $L_q$-norm loss function. The theoretical results are obtained in several cases with respect to the number of variable groups. This number can be fixed or dependent on the sample size $n$, with the…
$\ell_1$-penalized quantile regression is widely used for analyzing high-dimensional data with heterogeneity. It is now recognized that the $\ell_1$-penalty introduces non-negligible estimation bias, while a proper use of concave regularization may lead to estimators with refined convergence rates and oracle properties as the signal strengthens. Although folded concave penalized $M$-estimation with strongly convex loss functions have been well studied, the extant literature on quantile regression is relatively silent. The main difficulty is that the quantile loss is piecewise linear: it is non…
Variable-width confidence intervals in Gaussian regression and penalized maximum likelihood estimators
硬阈值、LASSO、自适应LASSO和SCAD点估计器已经被建议用于当回归参数向量的大多数分量被认为是零（稀疏型假设）时的线性回归上下文中。Potscher和Schneider，2010年，Electronic Journal of Statistics，考虑了固定宽度置信区间的属性，其中包括这些点估计量之一（对于所有可能的数据值）。他们认为一个正常的线性回归模型与正交回归，并表明，这些置信区间是长于标准置信区间（基于极大似然估计）时，这些点估计的调整参数选择，导致保守或一致的模型选择。我们将这种分析扩展到可变宽度置信区间的情况，其中包括这些点估计（对于所有可能的数据值）。与Potscher和Schneider的这些发现一致，我们发现这些置信区间与标准置信区间相比表现不佳，当选择这些点估计的调谐参数以导致一致的模型选择时。然而，当这些点估计的调整参数被选择，导致保守的模型选择，我们的结论不同于Potscher和Schneider。我们考虑Farchione和Kabaila，2008，Statistics & Probability Letters中的可变宽度置信区间，在稀疏性假设中存在信念的情况下，它比标准置信区间具有优势。这些可变宽度的置信区间，包括硬阈值，LASSO，自适应LASSO和SCAD估计（所有可能的数据值），这些估计的调谐参数被选择属于一个适当的区间。
In this paper, we introduce structured sparsity estimators in Generalized Linear Models. Structured sparsity estimators in the least squares loss are introduced by Stucky and van de Geer (2018) recently for fixed design and normal errors. We extend their results to debiased structured sparsity estimators with Generalized Linear Model based loss. Structured sparsity estimation means penalized loss functions with a possible sparsity structure used in the chosen norm. These include weighted group lasso, lasso and norms generated from convex cones. The significant difficulty is that it is not clea…
Erasmus School of Economics, Erasmus University Rotterdam（伊拉斯姆斯大学鹿特丹经济学院）；Tinbergen Institute（丁伯根研究所）；Geneva Finance Research Institute, University of Geneva（日内瓦大学日内瓦金融研究院）；Swiss Finance Institute（瑞士金融研究院）；Collegio Carlo Alberto, University of Turin（都灵大学卡洛阿尔贝托学院）
Nonconvex Penalized LAD Estimation in Partial Linear Models with DNNs: Asymptotic Analysis and Proximal Algorithms
Department of Applied Mathematics, The Hong Kong Polytechnic University（应用数学系，香港理工大学）；College of Computing, Data Science, and Society, University of California, Berkeley（计算、数据科学与社会学院，加州大学伯克利分校）
High-dimensional data can often display heterogeneity due to heteroscedastic variance or inhomogeneous covariate effects. Penalized quantile and expectile regression methods offer useful tools to detect heteroscedasticity in high-dimensional data. The former is computationally challenging due to the non-smooth nature of the check loss, and the latter is sensitive to heavy-tailed error distributions. In this paper, we propose and study (penalized) robust expectile regression (retire), with a focus on iteratively reweighted $\ell_1$-penalization which reduces the estimation bias from $\ell_1$-pe…
This paper investigates tradeoffs among optimization errors, statistical rates of convergence and the effect of heavy-tailed errors for high-dimensional robust regression with nonconvex regularization. When the additive errors in linear models have only bounded second moment, we show that iteratively reweighted $\ell_1$-penalized adaptive Huber regression estimator satisfies exponential deviation bounds and oracle properties, including the oracle convergence rate and variable selection consistency, under a weak beta-min condition. Computationally, we need as many as $O(\log s + \log\log d)$ it…
