We present a formula for the shrinkage factors of the Partial Least Squares regression estimator and deduce some of their properties, in particular the known fact that some of the factors are >1. We investigate the effect of shrinkage factors for the Mean Squared error of linear estimators and illustrate that we cannot extend the results to nonlinear estimators. In particular, shrinkage factors >1 do not automatically lead to a poorer Mean Squared Error. We investigate empirically the effect of bounding the the absolute value of the Partial Least Squares shrinkage factors by 1.
We show that prediction performance for global-local shrinkage regression can overcome two major difficulties of global shrinkage regression: (i) the amount of relative shrinkage is monotone in the singular values of the design matrix and (ii) the shrinkage is determined by a single tuning parameter. Specifically, we show that the horseshoe regression, with heavy-tailed component-specific local shrinkage parameters, in conjunction with a global parameter providing shrinkage towards zero, alleviates both these difficulties and consequently, results in an improved risk for prediction. Numerical …
The fuzzy linear regression (FLR) modeling was first proposed making use of linear programming and then followed by many improvements in a variety of ways. In almost all approaches changing the meters, objective function, and restrictions caused to improve the fuzzy measure of efficiencies (FMEs). In this paper, from a totally different viewpoint, we apply shrinkage estimation strategy to improve FMEs in the FLR modeling. By several illustrative examples, we demonstrate the superiority of the proposed estimation method. In this respect, we show fuzzy shrinkage estimates improve FMEs estimation…
During the past decade, shrinkage priors have received much attention in Bayesian analysis of high-dimensional data. This paper establishes the posterior consistency for high-dimensional linear regression with a class of shrinkage priors, which has a heavy and flat tail and allocates a sufficiently large probability mass in a very small neighborhood of zero. While enjoying its efficiency in posterior simulations, the shrinkage prior can lead to a nearly optimal posterior contraction rate and variable selection consistency as the spike-and-slab prior. Our numerical results show that under the p…
In this study, we propose shrinkage methods based on {\it generalized ridge regression} (GRR) estimation which is suitable for both multicollinearity and high dimensional problems with small number of samples (large $p$, small $n$). Also, it is obtained theoretical properties of the proposed estimators for Low/High Dimensional cases. Furthermore, the performance of the listed estimators is demonstrated by both simulation studies and real-data analysis, and compare its performance with existing penalty methods. We show that the proposed methods compare well to competing regularization technique…
We consider Bayesian shrinkage predictions for the Normal regression problem under the frequentist Kullback-Leibler risk function. Firstly, we consider the multivariate Normal model with an unknown mean and a known covariance. While the unknown mean is fixed, the covariance of future samples can be different from training samples. We show that the Bayesian predictive distribution based on the uniform prior is dominated by that based on a class of priors if the prior distributions for the covariance and future covariance matrices are rotation invariant. Then, we consider a class of priors for t…
We use Levy processes to generate joint prior distributions, and therefore penalty functions, for a location parameter as p grows large. This generalizes the class of local-global shrinkage rules based on scale mixtures of normals, illuminates new connections among disparate methods, and leads to new results for computing posterior means and modes under a wide class of priors. We extend this framework to large-scale regularized regression problems where p>n, and provide comparisons with other methodologies.
We propose a new penalized method for variable selection and estimation that explicitly incorporates the correlation patterns among predictors. This method is based on a combination of the minimax concave penalty and Laplacian quadratic associated with a graph as the penalty function. We call it the sparse Laplacian shrinkage (SLS) method. The SLS uses the minimax concave penalty for encouraging sparsity and Laplacian quadratic penalty for promoting smoothness among coefficients associated with the correlated predictors. The SLS has a generalized grouping property with respect to the graph rep…
Ranked set sampling (RSS) is used as a powerful data collection technique for situations where measuring the study variable requires a costly and/or tedious process while the sampling units can be ranked easily (e.g., osteoporosis research). In this paper, we develop ridge and Liu-type shrinkage estimators under RSS data from multiple observers to handle the collinearity problem in estimating coefficients of linear regression, stochastic restricted regression and logistic regression. Through extensive numerical studies, we show that shrinkage methods with the multi-observer RSS result in more …
The issue of honesty in constructing confidence sets arises in nonparametric regression. While optimal rate in nonparametric estimation can be achieved and utilized to construct sharp confidence sets, severe degradation of confidence level often happens after estimating the degree of smoothness. Similarly, for high-dimensional regression, oracle inequalities for sparse estimators could be utilized to construct sharp confidence sets. Yet the degree of sparsity itself is unknown and needs to be estimated, causing the honesty problem. To resolve this issue, we develop a novel method to construct …
Purdue University（普渡大学）；Department of Statistics（统计系）
We propose a variational Bayesian (VB) procedure for high-dimensional linear model inferences with heavy tail shrinkage priors, such as student-t prior. Theoretically, we establish the consistency of the proposed VB method and prove that under the proper choice of prior specifications, the contraction rate of the VB posterior is nearly optimal. It justifies the validity of VB inference as an alternative of Markov Chain Monte Carlo (MCMC) sampling. Meanwhile, comparing to conventional MCMC methods, the VB procedure achieves much higher computational efficiency, which greatly alleviates the comp…
For the regression model where the errors follow the elliptically contoured distribution (ECD), we consider the least squares (LS), restricted LS (RLS), preliminary test (PT), Stein-type shrinkage (S) and positive-rule shrinkage (PRS) estimators for the regression parameters. We compare the quadratic risks of the estimators to determine the relative dominance properties of the five estimators.
In this article we provide some nonnegative and positive estimators of the mean squared errors(MSEs) for shrinkage estimators of multivariate normal means. Proposed estimators are shown to improve on the uniformly minimum variance unbiased estimator(UMVUE) under a quadratic loss criterion. A similar improvement is also obtained for the estimators of the MSE matrices for shrinkage estimators. We also apply the proposed estimators of the MSE matrix to form confidence sets centered at shrinkage estimators and show their usefulness through numerical experiments.
We study general singular value shrinkage estimators in high-dimensional regression and classification, when the number of features and the sample size both grow proportionally to infinity. We allow models with general covariance matrices that include a large class of data generating distributions. As far as the implications of our results are concerned, we find exact asymptotic formulas for both the training and test errors in regression models fitted by gradient descent, which provides theoretical insights for early stopping as a regularization method. In addition, we propose a numerical met…
This paper constructs improved estimators of the means in the Gaussian saturated one-way layout with an ordinal factor. The least squares estimator for the mean vector in this saturated model is usually inadmissible. The hybrid shrinkage estimators of this paper exploit the possibility of slow variation in the dependence of the means on the ordered factor levels but do not assume it and respond well to faster variation if present. To motivate the development, candidate penalized least squares (PLS) estimators for the mean vector of a one-way layout are represented as shrinkage estimators relat…
High-dimensional data subject to heavy-tailed phenomena and heterogeneity are commonly encountered in various scientific fields and bring new challenges to the classical statistical methods. In this paper, we combine the asymmetric square loss and huber-type robust technique to develop the robust expectile regression for ultrahigh dimensional heavy-tailed heterogeneous data. Different from the classical huber method, we introduce two different tuning parameters on both sides to account for possibly asymmetry and allow them to diverge to reduce bias induced by the robust approximation. In the r…
This paper introduces a simple principle for robust high-dimensional statistical inference via an appropriate shrinkage on the data. This widens the scope of high-dimensional techniques, reducing the moment conditions from sub-exponential or sub-Gaussian distributions to merely bounded second or fourth moment. As an illustration of this principle, we focus on robust estimation of the low-rank matrix $Θ^*$ from the trace regression model $Y=Tr (Θ^{*T}X) +ε$. It encompasses four popular problems: sparse linear models, compressed sensing, matrix completion and multi-task regression. We propose to…
In this paper, we consider an estimation problem of the regression coefficients in multiple regression models with several unknown change-points. Under some realistic assumptions, we propose a class of estimators which includes as a special cases shrinkage estimators (SEs) as well as the unrestricted estimator (UE) and the restricted estimator (RE). We also derive a more general condition for the SEs to dominate the UE. To this end, we generalize some identities for the evaluation of the bias and risk functions of shrinkage-type estimators. As illustrative example, our method is applied to the…
We investigate the asymptotic behavior of posterior distributions of regression coefficients in high-dimensional linear models as the number of dimensions grows with the number of observations. We show that the posterior distribution concentrates in neighborhoods of the true parameter under simple sufficient conditions. These conditions hold under popular shrinkage priors given some sparsity assumptions.
We derive minimax generalized Bayes estimators of regression coefficients in the general linear model with spherically symmetric errors under invariant quadratic loss for the case of unknown scale. The class of estimators generalizes the class considered in Maruyama and Strawderman (2005) to include non-monotone shrinkage functions.
The exponential distribution is applied in a very wide variety of statistical procedures. Among the most prominent applications are those in the field of life testing and reliability theory. When there are two record samples available for estimating the scale parameter, a preliminary test is usually used to determine whether to pool the samples or use the individual sample. In this paper, the preliminary test estimator and shrinkage estimator are studied. The optimum level of significance for preliminary test estimation and the optimum values of shrinkage coefficient are obtained based on mini…
In this study, we consider preliminary test and shrinkage estimation strategies for quantile regression models. In classical Least Squares Estimation (LSE) method, the relationship between the explanatory and explained variables in the coordinate plane is estimated with a mean regression line. In order to use LSE, there are three main assumptions on the error terms showing white noise process of the regression model, also known as Gauss-Markov Assumptions, must be met: (1) The error terms have zero mean, (2) The variance of the error terms is constant and (3) The covariance between the errors …
This paper is speculated to propose a class of shrinkage estimators for shape parameter beta in failure censored samples from two-parameter Weibull distribution when some 'apriori' or guessed interval containing the parameter beta is available in addition to sample information and analyses their properties. Some estimators are generated from the proposed class and compared with the minimum mean squared error (MMSE) estimator. Numerical computations in terms of percent relative efficiency and absolute relative bias indicate that certain of these estimators substantially improve the MMSE estimat…
The James-Stein estimator is an estimator of the multivariate normal mean and dominates the maximum likelihood estimator (MLE) under squared error loss. The original work inspired great interest in developing shrinkage estimators for a variety of problems. Nonetheless, research on shrinkage estimation for manifold-valued data is scarce. In this paper, we propose shrinkage estimators for the parameters of the Log-Normal distribution defined on the manifold of $N \times N$ symmetric positive-definite matrices. For this manifold, we choose the Log-Euclidean metric as its Riemannian metric since i…
Shrinkage estimation usually reduces variance at the cost of bias. But when we care only about some parameters of a model, I show that we can reduce variance without incurring bias if we have additional information about the distribution of covariates. In a linear regression model with homoscedastic Normal noise, I consider shrinkage estimation of the nuisance parameters associated with control variables. For at least three control variables and exogenous treatment, I establish that the standard least-squares estimator is dominated with respect to squared-error loss in the treatment effect eve…
Data in non-Euclidean spaces are commonly encountered in many fields of Science and Engineering. For instance, in Robotics, attitude sensors capture orientation which is an element of a Lie group. In the recent past, several researchers have reported methods that take into account the geometry of Lie Groups in designing parameter estimation algorithms in nonlinear spaces. Maximum likelihood estimators (MLE) are quite commonly used for such tasks and it is well known in the field of statistics that Stein's shrinkage estimators dominate the MLE in a mean-squared sense assuming the observations a…
This paper considers a multiple regression model and compares, under full model hypothesis, analytically as well as by simulation, the performance characteristics of some popular penalty estimators such as ridge regression, LASSO, adaptive LASSO, SCAD, and elastic net versus Least Squares Estimator, restricted estimator, preliminary test estimator, and Stein-type estimators when the dimension of the parameter space is smaller than the sample space dimension. We find that RR uniformly dominates LSE, RE, PTE, SE and PRSE while LASSO, aLASSO, SCAD, and EN uniformly dominates LSE only. Further, it…
The two-stage least-squares (2SLS) estimator is known to be biased when its first-stage fit is poor. I show that better first-stage prediction can alleviate this bias. In a two-stage linear regression model with Normal noise, I consider shrinkage in the estimation of the first-stage instrumental variable coefficients. For at least four instrumental variables and a single endogenous regressor, I establish that the standard 2SLS estimator is dominated with respect to bias. The dominating IV estimator applies James-Stein type shrinkage in a first-stage high-dimensional Normal-means problem follow…
The negative multinomial distribution is a multivariate generalization of the negative binomial distribution. In this paper, we consider the problem of estimating an unknown matrix of probabilities on the basis of observations of negative multinomial variables under the standardized squared error loss. First, a general sufficient condition for a shrinkage estimator to dominate the UMVU estimator is derived and an empirical Bayes estimator satisfying the condition is constructed. Next, a hierarchical shrinkage prior is introduced, an associated Bayes estimator is shown to dominate the UMVU esti…
