In the framework of matrix valued observables with low rank means, Stein's unbiased risk estimate (SURE) can be useful for risk estimation and for tuning the amount of shrinkage towards low rank matrices. This was demonstrated by Candès et al. (2013) for singular value soft thresholding, which is a Lipschitz continuous estimator. SURE provides an unbiased risk estimate for an estimator whenever the differentiability requirements for Stein's lemma are satisfied. Lipschitz continuity of the estimator is sufficient, but it is emphasized that differentiability Lebesgue almost everywhere isn't. The reduced rank estimator, which gives the best approximation of the observation with a fixed rank, is an example of a discontinuous estimator for which Stein's lemma actually applies. This was observed by Mukherjee et al. (2015), but the proof was incomplete. This brief note gives a sufficient condition for Stein's lemma to hold for estimators with discontinuities, which is then shown to be fulfilled for a class of spectral function estimators including the reduced rank estimator. Singular value hard thresholding does, however, not satisfy the condition, and Stein's lemma does not apply to this estimator.
The Frechet mean is a useful description of location for a probability distribution on a metric space that is not necessarily a vector space. This article considers simultaneous estimation of multiple Frechet means from a decision-theoretic perspective, and in particular, the extent to which the unbiased estimator of a Frechet mean can be dominated by a generalization of the James-Stein shrinkage estimator. It is shown that if the metric space satisfies a non-positive curvature condition, then this generalized James-Stein estimator asymptotically dominates the unbiased estimator as the dimensi…
Stein's formula states that a random variable of the form $z^\top f(z) - \text{div} f(z)$ is mean-zero for functions $f$ with integrable gradient. Here, $\text{div} f$ is the divergence of the function $f$ and $z$ is a standard normal vector. This paper aims to propose a Second Order Stein formula to characterize the variance of such random variables for all functions $f(z)$ with square integrable gradient, and to demonstrate the usefulness of this formula in various applications. In the Gaussian sequence model, a consequence of Stein's formula is Stein's Unbiased Risk Estimate (SURE), an unbi…
Algorithms to solve variational regularization of ill-posed inverse problems usually involve operators that depend on a collection of continuous parameters. When these operators enjoy some (local) regularity, these parameters can be selected using the so-called Stein Unbiased Risk Estimate (SURE). While this selection is usually performed by exhaustive search, we address in this work the problem of using the SURE to efficiently optimize for a collection of continuous parameters of the model. When considering non-smooth regularizers, such as the popular l1-norm corresponding to soft-thresholdin…
Stein unbiased risk estimation is generalized twice, from the Gaussian shift model to nonparametric families of smooth densities, and from the quadratic risk to more general divergence type distances. The development relies on a connection with local proper scoring rules.
We investigate the problems of drift estimation for a shifted Brownian motion and intensity estimation for a Cox process on a finite interval $[0,T]$, when the risk is given by the energy functional associated to some fractional Sobolev space $H^1_0\subset W^{α,2}\subset L^2$. In both situations, Cramer-Rao lower bounds are obtained, entailing in particular that no unbiased estimators with finite risk in $H^1_0$ exist. By Malliavin calculus techniques, we also study super-efficient Stein type estimators (in the Gaussian case).
In this work, we construct a risk estimator for hard thresholding which can be used as a basis to solve the difficult task of automatically selecting the threshold. As hard thresholding is not even continuous, Stein's lemma cannot be used to get an unbiased estimator of degrees of freedom, hence of the risk. We prove that under a mild condition, our estimator of the degrees of freedom, although biased, is consistent. Numerical evidence shows that our estimator outperforms another biased risk estimator.
The problem of estimating a mean matrix of a multivariate complex normal distribution with an unknown covariance matrix is considered under an invariant loss function. By using complex versions of the Stein identity, the Stein-Haff identity, and calculus on eigenvalues, a formula is obtained for an unbiased estimate of the risk of an invariant class of estimators, from which several minimax shrinkage estimators are constructed.
Nearly all estimators in statistical prediction come with an associated tuning parameter, in one way or another. Common practice, given data, is to choose the tuning parameter value that minimizes a constructed estimate of the prediction error of the estimator; we focus on Stein's unbiased risk estimator, or SURE (Stein, 1981; Efron, 1986) which forms an unbiased estimate of the prediction error by augmenting the observed training error with an estimate of the degrees of freedom of the estimator. Parameter tuning via SURE minimization has been advocated by many authors, in a wide variety of pr…
We identify and correct an error in the paper "Excess Optimism: How Biased is the Apparent Error of an Estimator Tuned by SURE?" This correction allows new guarantees on the excess degrees of freedom--the bias in the error estimate of Stein's unbiased risk estimate (SURE) for an estimator tuned by directly minimizing the SURE criterion--for arbitrary SURE-tuned linear estimators. Oracle inequalities follow as a consequence of these results for such estimators.
We consider quasi-admissibility/inadmissibility of Stein-type shrinkage estimators of the mean of a multivariate normal distribution with covariance matrix an unknown multiple of the identity. Quasi-admissibility/inadmissibility is defined in terms of non-existence/existence of a solution to a differential inequality based on Stein's unbiased risk estimate (SURE). We find a sharp boundary between quasi-admissible and quasi-inadmissible estimators related to the optimal James-Stein estimator. We also find a class of priors related to the Strawderman class in the known variance case where the bo…
We study the effective degrees of freedom of the lasso in the framework of Stein's unbiased risk estimation (SURE). We show that the number of nonzero coefficients is an unbiased estimate for the degrees of freedom of the lasso--a conclusion that requires no special assumption on the predictors. In addition, the unbiased estimator is shown to be asymptotically consistent. With these results on hand, various model selection criteria--$C_p$, AIC and BIC--are available, which, along with the LARS algorithm, provide a principled and efficient approach to obtaining the optimal lasso fit with the co…
Many statistical studies are concerned with the analysis of observations organized in a matrix form whose elements are count data. When these observations are assumed to follow a Poisson or a multinomial distribution, it is of interest to focus on the estimation of either the intensity matrix (Poisson case) or the compositional matrix (multinomial case) when it is assumed to have a low rank structure. In this setting, it is proposed to construct an estimator minimizing the regularized negative log-likelihood by a nuclear norm penalty. Such an approach easily yields a low-rank matrix-valued est…
Shrinkage estimation usually reduces variance at the cost of bias. But when we care only about some parameters of a model, I show that we can reduce variance without incurring bias if we have additional information about the distribution of covariates. In a linear regression model with homoscedastic Normal noise, I consider shrinkage estimation of the nuisance parameters associated with control variables. For at least three control variables and exogenous treatment, I establish that the standard least-squares estimator is dominated with respect to squared-error loss in the treatment effect eve…
Let $X_1,\ldots,X_n$ be a random sample from the Gamma distribution with density $f(x)=λ^αx^{α-1}e^{-λx}/Γ(α)$, $x>0$, where both $α>0$ (the shape parameter) and $λ>0$ (the reciprocal scale parameter) are unknown. The main result shows that the uniformly minimum variance unbiased estimator (UMVUE) of the shape parameter, $α$, exists if and only if $n\geq 4$; moreover, it has finite variance if and only if $n\geq 6$. More precisely, the form of the UMVUE is given for all parametric functions $α$, $λ$, $1/α$ and $1/λ$. Furthermore, a highly efficient estimating procedure for the two-parameter Be…
Recently, the well known Liu estimator (Liu, 1993) is attracted researcher's attention in regression parameter estimation for an ill conditioned linear model. It is also argued that imposing sub-space hypothesis restriction on parameters improves estimation by shrinking toward non-sample information. Chang (2015) proposed the almost unbiased Liu estimator (AULE) in the binary logistic regression. In this article, some improved unbiased Liu type estimators, namely, restricted AULE, preliminary test AULE, Stein-type shrinkage AULE and its positive part for estimating the regression parameters in…
Using integration by parts on Gaussian space we construct a Stein Unbiased Risk Estimator (SURE) for the drift of Gaussian processes using their local and occupation times. By almost-sure minimization of the SURE risk of shrinkage estimators we derive an estimation and de-noising procedure for an input signal perturbed by a continuous-time Gaussian noise.
In this paper, we are interested in the classical problem of restoring data degraded by a convolution and the addition of a white Gaussian noise. The originality of the proposed approach is two-fold. Firstly, we formulate the restoration problem as a nonlinear estimation problem leading to the minimization of a criterion derived from Stein's unbiased quadratic risk estimate. Secondly, the deconvolution procedure is performed using any analysis and synthesis frames that can be overcomplete or not. New theoretical results concerning the calculation of the variance of the Stein's risk estimate ar…
Estimating a low rank matrix from its linear measurements is a problem of central importance in contemporary statistical analysis. The choice of tuning parameters for estimators remains an important challenge from a theoretical and practical perspective. To this end, Stein's Unbiased Risk Estimate (SURE) framework provides a well-grounded statistical framework for degrees of freedom estimation. In this paper, we use the SURE framework to obtain degrees of freedom estimates for a general class of spectral regularized matrix estimators, generalizing beyond the class of estimators that have been …
The recent thought-provoking paper by Hansen [2022, Econometrica] proved that the Gauss-Markov theorem continues to hold without the requirement that competing estimators are linear in the vector of outcomes. Despite the elegant proof, it was shown by the authors and other researchers that the main result in the earlier version of Hansen's paper does not extend the classic Gauss-Markov theorem because no nonlinear unbiased estimator exists under his conditions. To address the issue, Hansen [2022] added statements in the latest version with new conditions under which nonlinear unbiased estimato…
Stein-Rule Shrinkage for Stochastic Gradient Estimation in High Dimensions
Department of Statistics Faculty of Mathematical Sciences（统计学系数学科学学院）；Division of Computer Science Department of Applied Mathematics Faculty of Mathematical Sciences（计算机科学系应用数学系数学科学学院）
A data-driven block thresholding procedure for wavelet regression is proposed and its theoretical and numerical properties are investigated. The procedure empirically chooses the block size and threshold level at each resolution level by minimizing Stein's unbiased risk estimate. The estimator is sharp adaptive over a class of Besov bodies and achieves simultaneously within a small constant factor of the minimax risk over a wide collection of Besov Bodies including both the ``dense'' and ``sparse'' cases. The procedure is easy to implement. Numerical results show that it has superior finite sa…
We consider the problem of estimating a low-rank signal matrix from noisy measurements under the assumption that the distribution of the data matrix belongs to an exponential family. In this setting, we derive generalized Stein's unbiased risk estimation (SURE) formulas that hold for any spectral estimators which shrink or threshold the singular values of the data matrix. This leads to new data-driven spectral estimators, whose optimality is discussed using tools from random matrix theory and through numerical experiments. Under the spiked population model and in the asymptotic setting where t…
In this paper, we aim at recovering an unknown signal x0 from noisy L1measurements y=Phi*x0+w, where Phi is an ill-conditioned or singular linear operator and w accounts for some noise. To regularize such an ill-posed inverse problem, we impose an analysis sparsity prior. More precisely, the recovery is cast as a convex optimization program where the objective is the sum of a quadratic data fidelity term and a regularization term formed of the L1-norm of the correlations between the sought after signal and atoms in a given (generally overcomplete) dictionary. The L1-sparsity analysis prior is …
Stein's unbiased risk estimate (SURE) was proposed by Stein for the independent, identically distributed (iid) Gaussian model in order to derive estimates that dominate least-squares (LS). In recent years, the SURE criterion has been employed in a variety of denoising problems for choosing regularization parameters that minimize an estimate of the mean-squared error (MSE). However, its use has been limited to the iid case which precludes many important applications. In this paper we begin by deriving a SURE counterpart for general, not necessarily iid distributions from the exponential family.…
We consider the problem of combining data from observational and experimental sources to make causal conclusions. This problem is increasingly relevant, as the modern era has yielded passive collection of massive observational datasets in areas such as e-commerce and electronic health. These data may be used to supplement experimental data, which is frequently expensive to obtain. In Rosenman et al. (2018), we considered this problem under the assumption that all confounders were measured. Here, we relax the assumption of unconfoundedness. To derive combined estimators with desirable propertie…
From Cross-Validation to SURE: Asymptotic Risk of Tuned Regularized Estimators
Department of Economics, University of Pennsylvania（宾夕法尼亚大学经济学系）；Department of Economics, University of Oxford（牛津大学经济学系）；Department of Electrical Engineering and Computer Science, MIT（麻省理工学院电气工程与计算机科学系）
Risk Estimators for Choosing Regularization Parameters in Ill-Posed Problems - Properties and Limitations
University College London（伦敦大学学院）；Georg-August-Universität Göttingen（哥廷根大学）；University of Twente（特温特大学）；Ruhr-Universität Bochum（波鸿鲁尔大学）；Westfälische Wilhelms-Universität (WWU) Münster（明斯特大学）
