Rejoinder: The Dantzig selector: Statistical estimation when $p$ is much larger than $n$

The Dantzig selector: Statistical estimation when $p$ is much larger than $n$

In many important statistical applications, the number of variables or parameters $p$ is much larger than the number of observations $n$. Suppose then that we have observations $y=Xβ+z$, where $β\in\mathbf{R}^p$ is a parameter vector of interest, $X$ is a data matrix with possibly far fewer rows than columns, $n\ll p$, and the $z_i$'s are i.i.d. $N(0,σ^2)$. Is it possible to estimate $β$ reliably based on the noisy data $y$? To estimate $β$, we introduce a new estimator--we call it the Dantzig selector--which is a solution to the $\ell_1$-regularization problem \[\min_{\tilde{\b eta}\in\mathbf…

通过广义Dantzig选择器的可解释标量对图像线性回归模型

Interpretable Scalar-on-Image Linear Regression Models via the Generalized Dantzig Selector

提出广义Dantzig选择器，联合施加稀疏性和平滑性约束，提高标量对图像回归中系数函数的可解释性，并通过理论和实验验证其优势。

Adaptive post-Dantzig estimation and prediction for non-sparse "large $p$ and small $n$" models

For consistency (even oracle properties) of estimation and model prediction, almost all existing methods of variable/feature selection critically depend on sparsity of models. However, for ``large $p$ and small $n$" models sparsity assumption is hard to check and particularly, when this assumption is violated, the consistency of all existing estimations is usually impossible because working models selected by existing methods such as the LASSO and the Dantzig selector are usually biased. To attack this problem, we in this paper propose adaptive post-Dantzig estimation and model prediction. Her…

The Dantzig selector for diffusion processes with covariates

The Dantzig selector for a special parametric model of diffusion processes is studied in this paper. In our model, the diffusion coefficient is given as the exponential of the linear combination of other processes which are regarded as covariates. We propose an estimation procedure which is an adaptation of the Dantzig selector for linear regression models and prove the $l_q$ consistency of the estimator for all $q \in [1,\infty]$.

Covariate Selection in High-Dimensional Generalized Linear Models With Measurement Error

In many problems involving generalized linear models, the covariates are subject to measurement error. When the number of covariates p exceeds the sample size n, regularized methods like the lasso or Dantzig selector are required. Several recent papers have studied methods which correct for measurement error in the lasso or Dantzig selector for linear models in the p>n setting. We study a correction for generalized linear models based on Rosenbaum and Tsybakov's matrix uncertainty selector. By not requiring an estimate of the measurement error covariance matrix, this generalized matrix uncerta…

Parallelism, Uniqueness, and Large-Sample Asymptotics for the Dantzig Selector

The Dantzig selector (Candes and Tao, 2007) is a popular l1-regularization method for variable selection and estimation in linear regression. We present a very weak geometric condition on the observed predictors which is related to parallelism and, when satisfied, ensures the uniqueness of Dantzig selector estimators. The condition holds with probability 1, if the predictors are drawn from a continuous distribution. We discuss the necessity of this condition for uniqueness and also provide a closely related condition which ensures uniqueness of lasso estimators (Tibshirani, 1996). Large sample…

Simultaneous analysis of Lasso and Dantzig selector

We exhibit an approximate equivalence between the Lasso estimator and Dantzig selector. For both methods we derive parallel oracle inequalities for the prediction risk in the general nonparametric regression model, as well as bounds on the $\ell_p$ estimation loss for $1\le p\le 2$ in the linear model when the number of variables can be much larger than the sample size.

The Dantzig selector for a linear model of diffusion processes

In this paper, a linear model of diffusion processes with unknown drift and diagonal diffusion matrices is discussed. We will consider the estimation problems for unknown parameters based on the discrete time observation in high-dimensional and sparse settings. To estimate drift matrices, the Dantzig selector which was proposed by Candés and Tao in 2007 will be applied. Then, we will prove two types of consistency of the estimator of drift matrix; one is the consistency in the sense of $l_q$ norm for every $q \in [1,\infty]$ and the other is the variable selection consistency. Moreover, we wil…

Transductive versions of the LASSO and the Dantzig Selector

Transductive methods are useful in prediction problems when the training dataset is composed of a large number of unlabeled observations and a smaller number of labeled observations. In this paper, we propose an approach for developing transductive prediction procedures that are able to take advantage of the sparsity in the high dimensional linear regression. More precisely, we define transductive versions of the LASSO and the Dantzig Selector . These procedures combine labeled and unlabeled observations of the training dataset to produce a prediction for the unlabeled observations. We propose…

通过Dantzig选择器进行两步估计：高维参数的随机过程模型

Two step estimations via the Dantzig selector for models of stochastic processes with high-dimensional parameters

本文通过Dantzig选择器和两步估计方法，研究了高维参数随机过程模型的稀疏估计及渐近正态性。

Transductive versions of the LASSO and the Dantzig Selector

We consider the linear regression problem, where the number $p$ of covariates is possibly larger than the number $n$ of observations $(x_{i},y_{i})_{i\leq i \leq n}$, under sparsity assumptions. On the one hand, several methods have been successfully proposed to perform this task, for example the LASSO or the Dantzig Selector. On the other hand, consider new values $(x_{i})_{n+1\leq i \leq m}$. If one wants to estimate the corresponding $y_{i}$'s, one should think of a specific estimator devoted to this task, referred by Vapnik as a "transductive" estimator. This estimator may differ from an e…

The $l_q$ consistency of the Dantzig Selector for Cox's Proportional Hazards Model

The Dantzig selector for the proportional hazards model proposed by D.R. Cox is studied in a high-dimensional and sparse setting. We prove the $l_q$ consistency for all $q \geq 1$ of some estimators based on the compatibility factor, the weak cone invertibility factor, and the restricted eigenvalue for certain deterministic matrix which approximates the Hessian matrix of log partial likelihood. Our matrix conditions for these three factors are weaker than those of previous researches.

The Dantzig selector: Recovery of Signal via $\ell_1-α\ell_2$ Minimization

In the paper, we proposed the Dantzig selector based on the $\ell_{1}-α\ell_{2}$~$(0< α\leq1)$ minimization for the signal recovery. In the Dantzig selector, the constraint $\|{\bf A}^{\top}({\bf b}-{\bf A}{\bf x})\|_\infty \leq η$ for some small constant $η>0$ means the columns of ${\bf A}$ has very weakly correlated with the error vector ${\bf e}={\bf A}{\bf x}-{\bf b}$. First, recovery guarantees based on the restricted isometry property (RIP) are established for signals. Next, we propose the effective algorithm to solve the proposed Dantzig selector. Last, we illustrate the proposed model …

Variational Multiscale Nonparametric Regression: Smooth Functions

For the problem of nonparametric regression of smooth functions, we reconsider and analyze a constrained variational approach, which we call the MultIscale Nemirovski-Dantzig (MIND) estimator. This can be viewed as a multiscale extension of the Dantzig selector (\emph{Ann. Statist.}, 35(6): 2313--51, 2009) based on early ideas of Nemirovski (\emph{J. Comput. System Sci.}, 23:1--11, 1986). MIND minimizes a homogeneous Sobolev norm under the constraint that the multiresolution norm of the residual is bounded by a universal threshold. The main contribution of this paper is the derivation of conve…

Generalization of l1 constraints for high dimensional regression problems

We focus on the high dimensional linear regression $Y\sim\mathcal{N}(Xβ^{*},σ^{2}I_{n})$, where $β^{*}\in\mathds{R}^{p}$ is the parameter of interest. In this setting, several estimators such as the LASSO and the Dantzig Selector are known to satisfy interesting properties whenever the vector $β^{*}$ is sparse. Interestingly both of the LASSO and the Dantzig Selector can be seen as orthogonal projections of 0 into $\mathcal{DC}(s)=\{β\in\mathds{R}^{p},\|X'(Y-Xβ)\|_{\infty}\leq s\}$ - using an $\ell_{1}$ distance for the Dantzig Selector and $\ell_{2}$ for the LASSO. For a well chosen $s>0$, th…

High-dimensional stochastic optimization with the generalized Dantzig estimator

We propose a generalized version of the Dantzig selector. We show that it satisfies sparsity oracle inequalities in prediction and estimation. We consider then the particular case of high-dimensional linear regression model selection with the Huber loss function. In this case we derive the sup-norm convergence rate and the sign concentration property of the Dantzig estimators under a mutual coherence assumption on the dictionary.

A Remark on the Lasso and the Dantzig Selector

This article investigates a new parameter for the high-dimensional regression with noise: the distortion. This latter has attracted a lot of attention recently with the appearance of new deterministic constructions of 'almost'-Euclidean sections of the L1-ball. It measures how far is the intersection between the kernel of the design matrix and the unit L1-ball from an L2-ball. We show that the distortion holds enough information to derive oracle inequalities (i.e. a comparison to an ideal situation where one knows the s largest coefficients of the target) for the lasso and the Dantzig selector…

The Dantzig selector and sparsity oracle inequalities

Let \[Y_j=f_*(X_j)+ξ_j,\qquad j=1,...,n,\] where $X,X_1,...,X_n$ are i.i.d. random variables in a measurable space $(S,\mathcal{A})$ with distribution $Π$ and $ξ,ξ_1,... ,ξ_n$ are i.i.d. random variables with ${\mathbb{E}}ξ=0$ independent of $(X_1,...,X_n).$ Given a dictionary $h_1,...,h_N:S\mapsto{\mathbb{R}},$ let $f_λ:=\sum_{j=1}^Nλ_jh_j$, $λ=(λ_1,...,λ_N)\in{\mathbb{R}}^N.$ Given $\varepsilon>0,$ define \[\hatΛ_{\varepsilon}:=\Biggl\{\lam bda\in{\mathbb{R}}^N:\max_{1\leq k\leq N}\Biggl|n^{-1}\sum_{j=1}^n\big l(f_λ(X_j)-Y_j\bigr)h_k(X_j)\Biggr|\leq\varepsilon \Biggr\}\] and \[\hatλ:=\hatλ^{…

Discussion: A tale of three cousins: Lasso, L2Boosting and Dantzig

Discussion of ``The Dantzig selector: Statistical estimation when $p$ is much larger than $n$'' by Emmanuel Candes and Terence Tao [math/0506081]

A Theoretical Analysis of Sparse Recovery Stability of Dantzig Selector and LASSO

Dantzig selector (DS) and LASSO problems have attracted plenty of attention in statistical learning, sparse data recovery and mathematical optimization. In this paper, we provide a theoretical analysis of the sparse recovery stability of these optimization problems in more general settings and from a new perspective. We establish recovery error bounds for these optimization problems under a mild assumption called weak range space property of a transposed design matrix. This assumption is less restrictive than the well known sparse recovery conditions such as restricted isometry property (RIP),…

The Discrete Dantzig Selector: Estimating Sparse Linear Models via Mixed Integer Linear Optimization

We propose a novel high-dimensional linear regression estimator: the Discrete Dantzig Selector, which minimizes the number of nonzero regression coefficients subject to a budget on the maximal absolute correlation between the features and residuals. Motivated by the significant advances in integer optimization over the past 10-15 years, we present a Mixed Integer Linear Optimization (MILO) approach to obtain certifiably optimal global solutions to this nonconvex optimization problem. The current state of algorithmics in integer optimization makes our proposal substantially more computationally…

Optimal designs for Lasso and Dantzig selector using Expander Codes

We investigate the high-dimensional regression problem using adjacency matrices of unbalanced expander graphs. In this frame, we prove that the $\ell_{2}$-prediction error and the $\ell_{1}$-risk of the lasso and the Dantzig selector are optimal up to an explicit multiplicative constant. Thus we can estimate a high-dimensional target vector with an error term similar to the one obtained in a situation where one knows the support of the largest coordinates in advance. Moreover, we show that these design matrices have an explicit restricted eigenvalue. Precisely, they satisfy the restricted eige…

Cox's proportional hazards model with a high-dimensional and sparse regression parameter

This paper deals with the proportional hazards model proposed by D. R. Cox in a high-dimensional and sparse setting for a regression parameter. To estimate the regression parameter, the Dantzig selector is applied. The variable selection consistency of the Dantzig selector for the model will be proved. This property enables us to reduce the dimension of the parameter and to construct asymptotically normal estimators for the regression parameter and the cumulative baseline hazard function.

LASSO, Iterative Feature Selection and the Correlation Selector: Oracle Inequalities and Numerical Performances

We propose a general family of algorithms for regression estimation with quadratic loss. Our algorithms are able to select relevant functions into a large dictionary. We prove that a lot of algorithms that have already been studied for this task (LASSO and Group LASSO, Dantzig selector, Iterative Feature Selection, among others) belong to our family, and exhibit another particular member of this family that we call Correlation Selector in this paper. Using general properties of our family of algorithm we prove oracle inequalities for IFS, for the LASSO and for the Correlation Selector, and com…

Accuracy Assessment for High-dimensional Linear Regression

This paper considers point and interval estimation of the $\ell_q$ loss of an estimator in high-dimensional linear regression with random design. We establish the minimax rate for estimating the $\ell_{q}$ loss and the minimax expected length of confidence intervals for the $\ell_{q}$ loss of rate-optimal estimators of the regression vector, including commonly used estimators such as Lasso, scaled Lasso, square-root Lasso and Dantzig Selector. Adaptivity of the confidence intervals for the $\ell_{q}$ loss is also studied. Both the setting of known identity design covariance matrix and known no…

Regularized estimation of linear functionals of precision matrices for high-dimensional time series

This paper studies a Dantzig-selector type regularized estimator for linear functionals of high-dimensional linear processes. Explicit rates of convergence of the proposed estimator are obtained and they cover the broad regime from i.i.d. samples to long-range dependent time series and from sub-Gaussian innovations to those with mild polynomial moments. It is shown that the convergence rates depend on the degree of temporal dependence and the moment conditions of the underlying linear processes. The Dantzig-selector estimator is applied to the sparse Markowitz portfolio allocation and the opti…

高维数据中的测量误差

High-Dimensional Data with Measurement Error

针对高维回归中协变量存在测量误差的问题，综述并比较了岭回归、Lasso、Dantzig选择器和弹性网及其误差校正变体的性能，通过模拟和真实数据提供实践建议。

L1-恢复的精度保证

Accuracy guarantees for L1-recovery

本文提出两种基于L1最小化的稀疏信号恢复新方法，通过可验证的性能保证和优化边界，构建统计性质优于常用方法的估计器，并利用非欧几里得基追踪算法计算精度界。

Estimation and inference for high-dimensional non-sparse models

To successfully work on variable selection, sparse model structure has become a basic assumption for all existing methods. However, this assumption is questionable as it is hard to hold in most of cases and none of existing methods may provide consistent estimation and accurate model prediction in nons-parse scenarios. In this paper, we propose semiparametric re-modeling and inference when the linear regression model under study is possibly non-sparse. After an initial working model is selected by a method such as the Dantzig selector adopted in this paper, we re-construct a globally unbiased …

Rate Optimal Estimation and Confidence Intervals for High-dimensional Regression with Missing Covariates

Although a majority of the theoretical literature in high-dimensional statistics has focused on settings which involve fully-observed data, settings with missing values and corruptions are common in practice. We consider the problems of estimation and of constructing component-wise confidence intervals in a sparse high-dimensional linear regression model when some covariates of the design matrix are missing completely at random. We analyze a variant of the Dantzig selector [9] for estimating the regression model and we use a de-biasing argument to construct component-wise confidence intervals.…

Minimax Estimation of Partially-Observed Vector AutoRegressions

High-dimensional time series are a core ingredient of the statistical modeling toolkit, for which numerous estimation methods are known.But when observations are scarce or corrupted, the learning task becomes much harder.The question is: how much harder? In this paper, we study the properties of a partially-observed Vector AutoRegressive process, which is a state-space model endowed with a stochastic observation mechanism.Our goal is to estimate its sparse transition matrix, but we only have access to a small and noisy subsample of the state components.Interestingly, the sampling process itsel…

Thresholded Lasso for high dimensional variable selection and statistical estimation

Given $n$ noisy samples with $p$ dimensions, where $n \ll p$, we show that the multi-step thresholding procedure based on the Lasso -- we call it the {\it Thresholded Lasso}, can accurately estimate a sparse vector $β\in \R^p$ in a linear model $Y = X β+ ε$, where $X_{n \times p}$ is a design matrix normalized to have column $\ell_2$ norm $\sqrt{n}$, and $ε\sim N(0, σ^2 I_n)$. We show that under the restricted eigenvalue (RE) condition (Bickel-Ritov-Tsybakov 09), it is possible to achieve the $\ell_2$ loss within a logarithmic factor of the ideal mean square error one would achieve with an {\e…