We consider estimation of a multivariate normal mean vector under sum of squared error loss. We propose a new class of smooth estimators parameterized by αdominating the James-Stein estimator. The estimator for α=1 corresponds to the generalized Bayes estimator with respect to the harmonic prior. When αgoes to infinity, the estimator converges to the James-Stein positive-part estimator. Thus the class of our estimators is a bridge between the admissible estimator (α=1) and the inadmissible estimator (α=\infty). Although the estimators have quasi-admissibility which is a weaker optimality than admissibility, the problem of determining whether or not the estimator for α>1 admissible is still open.
We consider the nonparametric functional estimation of the drift of a Gaussian process via minimax and Bayes estimators. In this context, we construct superefficient estimators of Stein type for such drifts using the Malliavin integration by parts formula and superharmonic functionals on Gaussian space. Our results are illustrated by numerical simulations and extend the construction of James--Stein type estimators for Gaussian processes by Berger and Wolpert [J. Multivariate Anal. 13 (1983) 401--424].
The problem of estimating a normal covariance matrix is considered from a decision-theoretic point of view, where the dimension of the covariance matrix is larger than the sample size. This paper addresses not only the nonsingular case but also the singular case in terms of the covariance matrix. Based on James and Stein's minimax estimator and on an orthogonally invariant estimator, some classes of estimators are unifiedly defined for any possible ordering on the dimension, the sample size and the rank of the covariance matrix. Unified dominance results on such classes are provided under a St…
The Stein paradox has played an influential role in the field of high dimensional statistics. This result warns that the sample mean, classically regarded as the "usual estimator", may be suboptimal in high dimensions. The development of the James-Stein estimator, that addresses this paradox, has by now inspired a large literature on the theme of "shrinkage" in statistics. In this direction, we develop a James-Stein type estimator for the first principal component of a high dimension and low sample size data set. This estimator shrinks the usual estimator, an eigenvector of a sample covariance…
The estimation of the mean matrix of the multivariate normal distribution is addressed in the high dimensional setting. Efron-Morris-type linear shrinkage estimators based on ridge estimators for the precision matrix instead of the Moore-Penrose generalized inverse are considered, and the weights in the ridge-type linear shrinkage estimators are estimated in terms of minimizing the Stein unbiased risk estimators under the quadratic loss. It is shown that the ridge-type linear shrinkage estimators with the estimated weights are minimax, and that the estimated weights converge to the optimal wei…
A new class of minimax Stein-type shrinkage estimators of a multivariate normal mean is studied where the shrinkage factor is based on an l_p norm. The proposed estimators allow some but not all coordinates to be estimated by 0 thereby allow sparsity as well as minimaxity.
In the estimation of the mean matrix in a multivariate normal distribution, the generalized Bayes estimators with closed forms are provided, and the sufficient conditions for their minimaxity are derived relative to both matrix and scalar quadratic loss functions. The generalized Bayes estimators of the covariance matrix are also given with closed forms, and the dominance properties are discussed for the Stein loss function.
In this paper we consider the nonparametric functional estimation of the drift of Gaussian processes using Paley-Wiener and Karhunen-Loève expansions. We construct efficient estimators for the drift of such processes, and prove their minimaxity using Bayes estimators. We also construct superefficient estimators of Stein type for such drifts using the Malliavin integration by parts formula and stochastic analysis on Gaussian space, in which superharmonic functionals of the process paths play a particular role. Our results are illustrated by numerical simulations and extend the construction of J…
This article discusses estimation of a multivariate normal mean based on heteroscedastic observations. Under heteroscedasticity, estimators shrinking more on the coordinates with larger variances, seem desirable. Although they are not necessarily minimax in the ordinary sense, we show that such James-Stein type estimators can be ensemble minimax, minimax with respect to the ensemble risk, related to empirical Bayes perspective of Efron and Morris.
The Frechet mean is a useful description of location for a probability distribution on a metric space that is not necessarily a vector space. This article considers simultaneous estimation of multiple Frechet means from a decision-theoretic perspective, and in particular, the extent to which the unbiased estimator of a Frechet mean can be dominated by a generalization of the James-Stein shrinkage estimator. It is shown that if the metric space satisfies a non-positive curvature condition, then this generalized James-Stein estimator asymptotically dominates the unbiased estimator as the dimensi…
We consider the problem of estimating the error variance in a general linear model when the error distribution is assumed to be spherically symmetric, but not necessary Gaussian. In particular we study the case of a scale mixture of Gaussians including the particularly important case of the multivariate-t distribution. Under Stein's loss, we construct a class of estimators that improve on the usual best unbiased (and best equivariant) estimator. Our class has the interesting double robustness property of being simultaneously generalized Bayes (for the same generalized prior) and minimax over t…
This paper investigates estimation of the mean vector under invariant quadratic loss for a spherically symmetric location family with a residual vector with density of the form $ f(x,u)=η^{(p+n)/2}f(η\{\|x-θ\|^2+\|u\|^2\}) $, where $η$ is unknown. We show that the natural estimator $x$ is admissible for $p=1,2$. Also, for $p\geq 3$, we find classes of generalized Bayes estimators that are admissible within the class of equivariant estimators of the form $\{1-ξ(x/\|u\|)\}x$. In the Gaussian case, a variant of the James--Stein estimator, $[1-\{(p-2)/(n+2)\}/\{\|x\|^2/\|u\|^2+(p-2)/(n+2)+1\}]x$, …
In many statistical problems, stochastic signals can be represented as a sequence of noisy wavelet coefficients. In this paper, we develop general empirical Bayes methods for the estimation of true signal. Our estimators approximate certain oracle separable rules and achieve adaptation to ideal risks and exact minimax risks in broad collections of classes of signals. In particular, our estimators are uniformly adaptive to the minimum risk of separable estimators and the exact minimax risks simultaneously in Besov balls of all smoothness and shape indices, and they are uniformly superefficient …
Using integration by parts on Gaussian space we construct a Stein Unbiased Risk Estimator (SURE) for the drift of Gaussian processes using their local and occupation times. By almost-sure minimization of the SURE risk of shrinkage estimators we derive an estimation and de-noising procedure for an input signal perturbed by a continuous-time Gaussian noise.
This paper first strictly proved that the growth of the second moment of a large class of Gaussian processes is not greater than power function and the covariance matrix is strictly positive definite. Under these two conditions, the maximum likelihood estimators of the mean and variance of such classes of drift Gaussian process have strong consistency under broader growth of t_n. At the same time, the asymptotic normality of binary random vectors and the Berry-Esséen bound of estimators are obtained by using the Stein's method via Malliavian calculus.
We develop an adaptive monotone shrinkage estimator for regression models with the following characteristics: i) dense coefficients with small but important effects; ii) a priori ordering that indicates the probable predictive importance of the features. We capture both properties with an empirical Bayes estimator that shrinks coefficients monotonically with respect to their anticipated importance. This estimator can be rapidly computed using a version of Pool-Adjacent-Violators algorithm. We show that the proposed monotone shrinkage approach is competitive with the class of all Bayesian estim…
In the framework of matrix valued observables with low rank means, Stein's unbiased risk estimate (SURE) can be useful for risk estimation and for tuning the amount of shrinkage towards low rank matrices. This was demonstrated by Candès et al. (2013) for singular value soft thresholding, which is a Lipschitz continuous estimator. SURE provides an unbiased risk estimate for an estimator whenever the differentiability requirements for Stein's lemma are satisfied. Lipschitz continuity of the estimator is sufficient, but it is emphasized that differentiability Lebesgue almost everywhere isn't. The…
Shrinkage estimation usually reduces variance at the cost of bias. But when we care only about some parameters of a model, I show that we can reduce variance without incurring bias if we have additional information about the distribution of covariates. In a linear regression model with homoscedastic Normal noise, I consider shrinkage estimation of the nuisance parameters associated with control variables. For at least three control variables and exogenous treatment, I establish that the standard least-squares estimator is dominated with respect to squared-error loss in the treatment effect eve…
We investigate the problems of drift estimation for a shifted Brownian motion and intensity estimation for a Cox process on a finite interval $[0,T]$, when the risk is given by the energy functional associated to some fractional Sobolev space $H^1_0\subset W^{α,2}\subset L^2$. In both situations, Cramer-Rao lower bounds are obtained, entailing in particular that no unbiased estimators with finite risk in $H^1_0$ exist. By Malliavin calculus techniques, we also study super-efficient Stein type estimators (in the Gaussian case).
Motivated by questions about dense (non-sparse) signals in high-dimensional data analysis, we study the unconditional out-of-sample prediction error (predictive risk) associated with three popular linear estimators for high-dimensional linear models: ridge regression estimators, scalar multiples of the ordinary least squares (OLS) estimator (referred to as James-Stein shrinkage estimators), and marginal regression estimators. The results in this paper require no assumptions about sparsity and imply: (i) if prior information about the population predictor covariance is available, then the ridge…
Gradient information on the sampling distribution can be used to reduce the variance of Monte Carlo estimators via Stein's method. An important application is that of estimating an expectation of a test function along the sample path of a Markov chain, where gradient information enables convergence rate improvement at the cost of a linear system which must be solved. The contribution of this paper is to establish theoretical bounds on convergence rates for a class of estimators based on Stein's method. Our analysis accounts for (i) the degree of smoothness of the sampling distribution and test…
In this paper, we revisit the original ideas of Stein and propose an estimator of the intensity parameter of a homogeneous Poisson point process defined in $\R^d$ and observed in a bounded window. The procedure is based on a new general integration by parts formula for Poisson point processes. We show that our Stein estimator outperforms the maximum likelihood estimator in terms of mean squared error. In particular, we show that in many practical situations we have a gain larger than 30\%.
In this work, we construct a risk estimator for hard thresholding which can be used as a basis to solve the difficult task of automatically selecting the threshold. As hard thresholding is not even continuous, Stein's lemma cannot be used to get an unbiased estimator of degrees of freedom, hence of the risk. We prove that under a mild condition, our estimator of the degrees of freedom, although biased, is consistent. Numerical evidence shows that our estimator outperforms another biased risk estimator.
The problem of estimating a high-dimensional sparse vector $\boldsymbolθ \in \mathbb{R}^n$ from an observation in i.i.d. Gaussian noise is considered. The performance is measured using squared-error loss. An empirical Bayes shrinkage estimator, derived using a Bernoulli-Gaussian prior, is analyzed and compared with the well-known soft-thresholding estimator. We obtain concentration inequalities for the Stein's unbiased risk estimate and the loss function of both estimators. The results show that for large $n$, both the risk estimate and the loss function concentrate on deterministic values clo…
This paper considers the problem of estimating a high-dimensional vector of parameters $\boldsymbolθ \in \mathbb{R}^n$ from a noisy observation. The noise vector is i.i.d. Gaussian with known variance. For a squared-error loss function, the James-Stein (JS) estimator is known to dominate the simple maximum-likelihood (ML) estimator when the dimension $n$ exceeds two. The JS-estimator shrinks the observed vector towards the origin, and the risk reduction over the ML-estimator is greatest for $\boldsymbolθ$ that lie close to the origin. JS-estimators can be generalized to shrink the data towards…
The problem of estimating the shift (or, equivalently, the center of symmetry) of an unknown symmetric and periodic function $f$ observed in Gaussian white noise is considered. Using the blockwise Stein method, a penalized profile likelihood with a data-driven penalization is introduced so that the estimator of the center of symmetry is defined as the maximizer of the penalized profile likelihood. This estimator has the advantage of being independent of the functional class to which the signal $f$ si assumed to belong and, furthermore, is shown to be semiparametrically adaptive and efficient. …
In a remarkable series of papers beginning in 1956, Charles Stein set the stage for the future development of minimax shrinkage estimators of a multivariate normal mean under quadratic loss. More recently, parallel developments have seen the emergence of minimax shrinkage estimators of multivariate normal predictive densities under Kullback--Leibler risk. We here describe these parallels emphasizing the focus on Bayes procedures and the derivation of the superharmonic conditions for minimaxity as well as further developments of new minimax shrinkage predictive density estimators including mult…
We are interested in the statistical linear inverse problem $Y=Af+εξ$, where $A$ denotes a compact operator and $εξ$ a stochastic noise. In a first time, we investigate the link between some threshold estimators and the risk hull point of view introduced in (5). The penalized blockwise Stein's rule plays a central role in this study. In particular, this estimator may be considered as a risk hull minimization method, provided the penalty is well-chosen. Using this perspective, we study the properties of the threshold and propose an admissible range for the penalty leading to accurate results. W…
Imperial College London（帝国理工学院）；University College London（伦敦大学学院）；University of Cambridge（剑桥大学）；Microsoft Research（微软研究院）
When maximum likelihood estimation is infeasible, one often turns to score matching, contrastive divergence, or minimum probability flow to obtain tractable parameter estimates. We provide a unifying perspective of these techniques as minimum Stein discrepancy estimators, and use this lens to design new diffusion kernel Stein discrepancy (DKSD) and diffusion score matching (DSM) estimators with complementary strengths. We establish the consistency, asymptotic normality, and robustness of DKSD and DSM estimators, then derive stochastic Riemannian gradient descent algorithms for their efficient …
We consider quasi-admissibility/inadmissibility of Stein-type shrinkage estimators of the mean of a multivariate normal distribution with covariance matrix an unknown multiple of the identity. Quasi-admissibility/inadmissibility is defined in terms of non-existence/existence of a solution to a differential inequality based on Stein's unbiased risk estimate (SURE). We find a sharp boundary between quasi-admissible and quasi-inadmissible estimators related to the optimal James-Stein estimator. We also find a class of priors related to the Strawderman class in the known variance case where the bo…
We study conditions under which treatment effect estimators constructed under the no-interference assumption in randomized experiments are asymptotically normal in the presence of interference. We prove that the standard Horvitz-Thompson estimator is asymptotically normal under a restricted interference condition characterized by limiting the degree of the dependency graph. The amount of interference is allowed to grow with the population size. We then provide a central limit theorem for the difference-in-means estimator that can handle interference that exists between all pairs of units, prov…
MPI for Intelligent Systems（马克斯·普朗克智能系统研究所）；The Institute of Statistical Mathematics（统计数学研究所）；University of Cambridge（剑桥大学）；University College London（伦敦大学学院）
A mean function in reproducing kernel Hilbert space, or a kernel mean, is an important part of many applications ranging from kernel principal component analysis to Hilbert-space embedding of distributions. Given finite samples, an empirical average is the standard estimate for the true kernel mean. We show that this estimator can be improved via a well-known phenomenon in statistics called Stein's phenomenon. After consideration, our theoretical analysis reveals the existence of a wide class of estimators that are better than the standard. Focusing on a subset of this class, we propose effici…
In this work, the estimation of the multivariate normal mean by different classes of shrinkage estimators is investigated. The risk associated with the balanced loss function is used to compare two estimators. We start by considering estimators that generalize the James-Stein estimator and show that these estimators dominate the maximum likelihood estimator (MLE), therefore are minimax, when the shrinkage function satisfies some conditions. Then, we treat estimators of polynomial form and prove the increase of the degree of the polynomial allows us to build a better estimator from the one prev…
The problem of estimating a mean matrix of a multivariate complex normal distribution with an unknown covariance matrix is considered under an invariant loss function. By using complex versions of the Stein identity, the Stein-Haff identity, and calculus on eigenvalues, a formula is obtained for an unbiased estimate of the risk of an invariant class of estimators, from which several minimax shrinkage estimators are constructed.
In this study, we consider preliminary test and shrinkage estimation strategies for quantile regression models. In classical Least Squares Estimation (LSE) method, the relationship between the explanatory and explained variables in the coordinate plane is estimated with a mean regression line. In order to use LSE, there are three main assumptions on the error terms showing white noise process of the regression model, also known as Gauss-Markov Assumptions, must be met: (1) The error terms have zero mean, (2) The variance of the error terms is constant and (3) The covariance between the errors …
Estimating a low rank matrix from its linear measurements is a problem of central importance in contemporary statistical analysis. The choice of tuning parameters for estimators remains an important challenge from a theoretical and practical perspective. To this end, Stein's Unbiased Risk Estimate (SURE) framework provides a well-grounded statistical framework for degrees of freedom estimation. In this paper, we use the SURE framework to obtain degrees of freedom estimates for a general class of spectral regularized matrix estimators, generalizing beyond the class of estimators that have been …
Algorithms to solve variational regularization of ill-posed inverse problems usually involve operators that depend on a collection of continuous parameters. When these operators enjoy some (local) regularity, these parameters can be selected using the so-called Stein Unbiased Risk Estimate (SURE). While this selection is usually performed by exhaustive search, we address in this work the problem of using the SURE to efficiently optimize for a collection of continuous parameters of the model. When considering non-smooth regularizers, such as the popular l1-norm corresponding to soft-thresholdin…
We consider estimating the parametric components of semi-parametric multiple index models in a high-dimensional and non-Gaussian setting. Such models form a rich class of non-linear models with applications to signal processing, machine learning and statistics. Our estimators leverage the score function based first and second-order Stein's identities and do not require the covariates to satisfy Gaussian or elliptical symmetry assumptions common in the literature. Moreover, to handle score functions and responses that are heavy-tailed, our estimators are constructed via carefully thresholding t…
We consider the linear regression problem of estimating an unknown, deterministic parameter vector based on measurements corrupted by colored Gaussian noise. We present and analyze blind minimax estimators (BMEs), which consist of a bounded parameter set minimax estimator, whose parameter set is itself estimated from measurements. Thus, one does not require any prior assumption or knowledge, and the proposed estimator can be applied to any linear regression problem. We demonstrate analytically that the BMEs strictly dominate the least-squares estimator, i.e., they achieve lower mean-squared er…
