Heavy-tailed error distributions and predictors with anomalous values are ubiquitous in high-dimensional regression problems and can seriously jeopardize the validity of statistical analyses if not properly addressed. For more reliable estimation under these adverse conditions, we propose a new robust regularized estimator for simultaneous variable selection and coefficient estimation. This estimator, called adaptive PENSE, possesses the oracle property without prior knowledge of the scale of the residuals and without any moment conditions on the error distribution. The proposed estimator gives reliable results even under very heavy-tailed error distributions and aberrant contamination in the predictors or residuals. Importantly, even in these challenging settings variable selection by adaptive PENSE remains stable. Numerical studies on simulated and real data sets highlight superior finite-sample performance in a vast range of settings compared to other robust regularized estimators in the case of contaminated samples and competitiveness compared to classical regularized estimators in clean samples.
In this paper, we propose a novel variable selection approach in the framework of high-dimensional linear models where the columns of the design matrix are highly correlated. It consists in rewriting the initial high-dimensional linear model to remove the correlation between the columns of the design matrix and in applying a generalized Elastic Net criterion since it can be seen as an extension of the generalized Lasso. The properties of our approach called gEN (generalized Elastic Net) are investigated both from a theoretical and a numerical point of view. More precisely, we provide a new con…
Fully robust versions of the elastic net estimator are introduced for linear and logistic regression. The algorithms to compute the estimators are based on the idea of repeatedly applying the non-robust classical estimators to data subsets only. It is shown how outlier-free subsets can be identified efficiently, and how appropriate tuning parameters for the elastic net penalties can be selected. A final reweighting step improves the efficiency of the estimators. Simulation studies compare with non-robust and other competing robust estimators and reveal the superiority of the newly proposed met…
Penalized linear regression is of fundamental importance in high-dimensional statistics and has been routinely used to regress a response on a high-dimensional set of predictors. In many scientific applications, there exists external information that encodes the predictive power and sparsity structure of the predictors. In this article, we propose the Structure Adaptive Elastic-Net (SA-Enet), which provides a new framework for incorporating potentially useful side information into a penalized regression. The basic idea is to translate the external information into different penalization streng…
This paper studies the multi-task high-dimensional linear regression models where the noise among different tasks is correlated, in the moderately high dimensional regime where sample size $n$ and dimension $p$ are of the same order. Our goal is to estimate the covariance matrix of the noise random vectors, or equivalently the correlation of the noise variables on any pair of two tasks. Treating the regression coefficients as a nuisance parameter, we leverage the multi-task elastic-net and multi-task lasso estimators to estimate the nuisance. By precisely understanding the bias of the squared …
We introduce the arbitrary rectangle-range generalized elastic net penalty method, abbreviated to ARGEN, for performing constrained variable selection and regularization in high-dimensional sparse linear models. As a natural extension of the nonnegative elastic net penalty method, ARGEN is proved to have variable selection consistency and estimation consistency under some conditions. The asymptotic behavior in distribution of the ARGEN estimators have been studied. We also propose an algorithm called MU-QP-RR-W-$l_1$ to efficiently solve ARGEN. By conducting simulation study we show that ARGEN…
When it comes to datasets with a tremendous amount of predictors, variable reduction techniques such as PCA or FA are often used. In this paper, the elastic net, which lies in between the LASSO method and ridge regression, is used as a variable reduction technique followed by further analysis with multivariate regression. Specifically, a messy only dataset is used to show how it can be 'tidied' up and broken down into sensible subsets using the aforementioned method.
The abundance of high-dimensional data in the modern sciences has generated tremendous interest in penalized estimators such as the lasso, scaled lasso, square-root lasso, elastic net, and many others. In this paper, we establish a general oracle inequality for prediction in high-dimensional linear regression with such methods. Since the proof relies only on convexity and continuity arguments, the result holds irrespective of the design matrix and applies to a wide range of penalized estimators. Overall, the bound demonstrates that generic estimators can provide consistent prediction with any …
Shrinkage estimators that possess the ability to produce sparse solutions have become increasingly important to the analysis of today's complex datasets. Examples include the LASSO, the Elastic-Net and their adaptive counterparts. Estimation of penalty parameters still presents difficulties however. While variable selection consistent procedures have been developed, their finite sample performance can often be less than satisfactory. We develop a new strategy for variable selection using the adaptive LASSO and adaptive Elastic-Net estimators with $p_n$ diverging. The basic idea first involves …
We propose a robust elastic net (REN) model for high-dimensional sparse regression and give its performance guarantees (both the statistical error bound and the optimization bound). A simple idea of trimming the inner product is applied to the elastic net model. Specifically, we robustify the covariance matrix by trimming the inner product based on the intuition that the trimmed inner product can not be significant affected by a bounded number of arbitrarily corrupted points (outliers). The REN model can also derive two interesting special cases: robust Lasso and robust soft thresholding. Comp…
Survival month for non-small lung cancer patients depend upon which stage of lung cancer is present. Our aim is to identify smoking specific gene expression biomarkers in the prognosis of lung cancer patients. In this paper, we introduce the network elastic net, a generalization of network lasso that allows for simultaneous clustering and regression on graphs. In Network elastic net, we consider similar patients based on smoking cigarettes per year to form the network. We then further find the suitable cluster among patients based on coefficients of genes having different survival month struct…
Variable Selection for Survival Data with A Class of Adaptive Elastic Net Techniques
加速失效时间（AFT）模型已被证明在许多情况下是有用的，尽管重删失（例如在癌症存活中）和高维度（例如在微阵列数据中）导致模型拟合和模型选择的困难。我们提出了新的方法，删失数据的变量选择，AFT模型优化使用正则化加权最小二乘的基础上。正则化技术在两种建议的弹性网类型方法下使用L1和L2范数惩罚的混合。一种是自适应弹性网，另一种是加权弹性网。这些方法分别扩展了Ghosh（2007）和Hong and Zhang（2010）提出的原始方法。我们还扩展了这两个建议的方法，通过增加审查意见作为约束到他们的模型优化框架。这些方法在微阵列上进行了评估，并通过模拟进行了评估。我们比较了这些方法与其他六种变量选择技术的性能-三种通常用于删失数据，其他三种是用于高维数据的基于相关性的贪婪方法。
Determining the mean shape of a collection of curves is not a trivial task, in particular when curves are only irregularly/sparsely sampled at discrete points. We newly propose an elastic full Procrustes mean of shapes of (oriented) plane curves, which are considered equivalence classes of parameterized curves with respect to translation, rotation, scale, and re-parameterization (warping), based on the square-root-velocity (SRV) framework. Identifying the real plane with the complex numbers, we establish a connection to covariance estimation in irregular/sparse functional data analysis. We int…
In the context of undirected Gaussian graphical models, we introduce three estimators based on elastic net penalty for the underlying dependence graph. Our goal is to estimate the sparse precision matrix, from which to retrieve both the underlying conditional dependence graph and the partial correlation graph. The first estimator is derived from the direct penalization of the precision matrix in the likelihood function, while the second from using conditional penalized regressions to estimate the precision matrix. Finally, the third estimator relies on a 2-stages procedure that estimates the e…
In real applications of the linear model, the explanatory variables are very often naturally grouped, the most common example being the multivariate variance analysis. In the present paper, a quantile model with structure group is considered, the number of groups can diverge with sample size. We introduce and study the adaptive elastic-net group estimator, for improving the parameter estimation accuracy. This method allows automatic selection, with a probability converging to one, of significant groups and further the non zero parameter estimators are asymptotically normal. The convergence rat…
We propose a new 2-stage procedure that relies on the elastic net penalty to estimate a network based on partial correlations when data are heavy-tailed. The new estimator allows to consider the lasso penalty as a special case. Using Monte Carlo simulations, we test the performance on several underlying network structures and four different multivariate distributions: Gaussian, t-Student with 3 and 20 degrees of freedom and contaminated Gaussian. Simulation analysis shows that the 2-stage estimator performs best for heavy-tailed data and it is also robust to distribution misspecification, both…
We propose a test-based elastic integrative analysis of the randomized trial and real-world data to estimate treatment effect heterogeneity with a vector of known effect modifiers. When the real-world data are not subject to bias, our approach combines the trial and real-world data for efficient estimation. Utilizing the trial design, we construct a test to decide whether or not to use real-world data. We characterize the asymptotic distribution of the test-based estimator under local alternatives. We provide a data-adaptive procedure to select the test threshold that promises the smallest mea…
Nowadays, clinical research routinely uses omics data, such as gene expression, for predicting clinical outcomes or selecting markers. Additionally, so-called co-data are often available, providing complementary information on the covariates, like p-values from previously published studies or groups of genes corresponding to pathways. Elastic net penalisation is widely used for prediction and covariate selection. Group-adaptive elastic net penalisation learns from co-data to improve the prediction and covariate selection, by penalising important groups of covariates less than other groups. Exi…
A robust and sparse estimator for multinomial regression is proposed for high dimensional data. Robustness of the estimator is achieved by trimming the observations, and sparsity of the estimator is obtained by the elastic net penalty, which is a mixture of $L_1$ and $L_2$ penalties. From this point of view, the proposed estimator is an extension of the enet-LTS estimator \citep{Kurnaz18} for linear and logistic regression to the multinomial regression setting. After introducing an algorithm for its computation, a simulation study is conducted to show the performance in comparison to the non-r…
This paper develops theory for feasible estimators of finite-dimensional parameters identified by general conditional quantile restrictions, under much weaker assumptions than previously seen in the literature. This includes instrumental variables nonlinear quantile regression as a special case. More specifically, we consider a set of unconditional moments implied by the conditional quantile restrictions, providing conditions for local identification. Since estimators based on the sample moments are generally impossible to compute numerically in practice, we study feasible estimators based on …
The lasso and elastic net are popular regularized regression models for supervised learning. Friedman, Hastie, and Tibshirani (2010) introduced a computationally efficient algorithm for computing the elastic net regularization path for ordinary least squares regression, logistic regression and multinomial logistic regression, while Simon, Friedman, Hastie, and Tibshirani (2011) extended this work to Cox models for right-censored data. We further extend the reach of the elastic net-regularized regression to all generalized linear model families, Cox models with (start, stop] data and strata, an…
Adaptive elastic net and Separate Selection from Least Squares for ultra-high dimensional regression models
研究了超高维稀疏线性回归模型中自适应弹性网的渐近性质，提出了一种新的提高预测精度的方法--SSLS（Separate Selection from Least Squares）方法.此外，我们还证明了SSLS在理论和实证方面都具有优越的性能。本文证明了自适应弹性网选错变量的概率在很少的条件下可以指数衰减。不可表示条件或类似的约束在我们的证明中是不必要的。我们得到了精确的偏差和均方误差（MSE）的界限，这两者都依赖于参数的选择，也表明存在的自适应弹性网的渐近正态性的偏差。此外，仿真和实证部分都表明，惩罚最小二乘的预测精度需要进一步提高。因此，我们提出了SSLS来改善预测。本文采用自适应弹性网，先选取变量，将高维降维。在第二步中，基于OLS估计构造系数。我们发现，SSLS的偏差可以以指数速率衰减。而且，MSE衰减到零。最后，我们证明了SSLS的变量选择一致性蕴涵了SSLS的渐近正态性。仿真结果表明了SSLS、自适应弹性网和其它惩罚最小二乘算法的性能。实证部分结合其他方法研究了股票市场的指数跟踪问题。
