The paper concerns small-area estimation in the Fay-Herriot type area-level model with random dispersions, which models the case that the sampling errors change from area to area. The resulting Bayes estimator shrinks both means and variances, but needs numerical computation to provide the estimates. In this paper, an approximated empirical Bayes (AEB) estimator with a closed form is suggested. The model parameters are estimated via the moment method, and the mean squared error of the AEB is estimated via the single parametric bootstrap method. The benchmarked estimator and a second-order unbiased estimator of the mean squared error are also derived.
In prediction problems with more predictors than observations, it can sometimes be helpful to use a joint probability model, $π(Y,X)$, rather than a purely conditional model, $π(Y \mid X)$, where $Y$ is a scalar response variable and $X$ is a vector of predictors. This approach is motivated by the fact that in many situations the marginal predictor distribution $π(X)$ can provide useful information about the parameter values governing the conditional regression. However, under very mild misspecification, this marginal distribution can also lead conditional inferences astray. Here, we explore t…
Designed gene expression micro-array experiments, consisting of several treatment levels with a number of replicates per level, are analyzed by applying simple tests for group differences at the per gene level. The gene level statistics are sorted and a criterion for selecting important genes which takes into account multiplicity is applied. A caveat arises in that true signals (genes truly over or under expressed) are "competing" with fairly large type I error signals. False positives near the top of a sorted list can occur when genes having very small fold-change are compensated by small eno…
Constrained approaches to maximum likelihood estimation in the context of finite mixtures of normals have been presented in the literature. A fully data-dependent constrained method for maximum likelihood estimation of clusterwise linear regression is proposed, which extends previous work in equivariant data-driven estimation of finite mixtures of Gaussians for classification. The method imposes plausible bounds on the component variances, based on a target value estimated from the data, which we take to be the homoscedastic variance. Nevertheless, the present work does not only focus on class…
Amazon Web Services（亚马逊网络服务公司）；University of Southern California（南加州大学）；Los Angeles, CA, USA（美国加利福尼亚州洛杉矶）
Estimation of uncertainty in deep learning models is of vital importance, especially in medical imaging, where reliance on inference without taking into account uncertainty could lead to misdiagnosis. Recently, the probabilistic Variational AutoEncoder (VAE) has become a popular model for anomaly detection in applications such as lesion detection in medical images. The VAE is a generative graphical model that is used to learn the data distribution from samples and then generate new samples from this distribution. By training on normal samples, the VAE can be used to detect inputs that deviate …
We develop and evaluate point and interval estimates for the random effects $θ_i$, having made observations $y_i|θ_i\stackrel{\m athit{ind}}{\sim}N[θ_i,V_i],i=1,...,k$ that follow a two-level Normal hierarchical model. Fitting this model requires assessing the Level-2 variance $A\equiv\operatorname {Var}(θ_i)$ to estimate shrinkages $B_i\equiv V_i/(V_i+A)$ toward a (possibly estimated) subspace, with $B_i$ as the target because the conditional means and variances of $θ_i$ depend linearly on $B_i$, not on $A$. Adjustment for density maximization, ADM, can do the fitting for any smooth prior on …
The widespread availability of high-dimensional biological data has made the simultaneous screening of many biological characteristics a central problem in computational biology and allied sciences. While the dimensionality of such datasets continues to grow, so too does the complexity of biomarker identification from exposure patterns in health studies measuring baseline confounders; moreover, doing so while avoiding model misspecification remains an issue only partially addressed. Efficient estimators capable of incorporating flexible, data adaptive regression techniques in estimating releva…
While shrinkage is essential in high-dimensional settings, its use for low-dimensional regression-based prediction has been debated. It reduces variance, often leading to improved prediction accuracy. However, it also inevitably introduces bias, which may harm two other measures of predictive performance: calibration and coverage of confidence intervals. Much of the criticism stems from the usage of standard shrinkage methods, such as lasso and ridge with a single, cross-validated penalty. Our aim is to show that readily available alternatives can strongly improve predictive performance, in te…
Heavy-tailed continuous shrinkage priors, such as the horseshoe prior, are widely used for sparse estimation problems. However, there is limited work extending these priors to predictors with grouping structures. Of particular interest in this article, is regression coefficient estimation where pockets of high collinearity in the covariate space are contained within known covariate groupings. To assuage variance inflation due to multicollinearity we propose the group inverse-gamma gamma (GIGG) prior, a heavy-tailed prior that can trade-off between local and group shrinkage in a data adaptive f…
When shrinking a covariance matrix towards (a multiple) of the identity matrix, the trace of the covariance matrix arises naturally as the optimal scaling factor for the identity target. The trace also appears in other context, for example when measuring the size of a matrix or the amount of uncertainty. Of particular interest is the case when the dimension of the covariance matrix is large. Then the problem arises that the sample covariance matrix is singular if the dimension is larger than the sample size. Another issue is that usually the estimation has to based on correlated time series da…
In this paper we propose an optimal predictor of a random variable that has either an infinite mean or an infinite variance. The method consists of transforming the random variable such that the transformed variable has a finite mean and finite variance. The proposed predictor is a generalized arithmetic mean which is similar to the notion of certainty price in utility theory. Typically, the transformation consists of a parametric family of bijections, in which case the parameter might be chosen to minimize the prediction error in the transformed coordinates. The statistical properties of the …
Consider estimating the n by p matrix of means of an n by p matrix of independent normally distributed observations with constant variance, where the performance of an estimator is judged using a p by p matrix quadratic error loss function. A matrix version of the James-Stein estimator is proposed, depending on a tuning constant. It is shown to dominate the usual maximum likelihood estimator for some choices of of the tuning constant when n is greater than or equal to 3. This result also extends to other shrinkage estimators and settings.
In this paper, we apply shrinkage strategies to estimate regression coefficients efficiently for the high-dimensional multiple regression model, where the number of samples is smaller than the number of predictors. We assume in the sparse linear model some of the predictors have very weak influence on the response of interest. We propose to shrink estimators more than usual. Specifically, we use integrated estimation strategies in sub and full models and shrink the integrated estimators by incorporating a bounded measurable function of some weights. The exhibited double shrunken estimators imp…
This paper considers a high-dimensional linear regression problem where there are complex correlation structures among predictors. We propose a graph-constrained regularization procedure, named Sparse Laplacian Shrinkage with the Graphical Lasso Estimator (SLS-GLE). The procedure uses the estimated precision matrix to describe the specific information on the conditional dependence pattern among predictors, and encourages both sparsity on the regression model and the graphical model. We introduce the Laplacian quadratic penalty adopting the graph information, and give detailed discussions on th…
Parameter shrinkage applied optimally can always reduce error and projection variances from those of maximum likelihood estimation. Many variables that actuaries use are on numerical scales, like age or year, which require parameters at each point. Rather than shrinking these towards zero, nearby parameters are better shrunk towards each other. Semiparametric regression is a statistical discipline for building curves across parameter classes using shrinkage methodology. It is similar to but more parsimonious than cubic splines. We introduce it in the context of Bayesian shrinkage and apply it …
Stein-Rule Shrinkage for Stochastic Gradient Estimation in High Dimensions
Department of Statistics Faculty of Mathematical Sciences（统计学系数学科学学院）；Division of Computer Science Department of Applied Mathematics Faculty of Mathematical Sciences（计算机科学系应用数学系数学科学学院）
The known connection between shrinkage estimation, empirical Bayes, and mixed effects models is explored and applied to balanced and unbalanced designs in which the responses are correlated. As an illustration, a mixed model is proposed for predicting the outcome of English Premier League games that takes into account both home and away team effects. Results based on empirical best linear unbiased predictors obtained from fitting mixed linear models are compared with fully Bayesian predictors that utilize prior information from the previous season.
Time-varying parameter (TVP) models are very flexible in capturing gradual changes in the effect of a predictor on the outcome variable. However, in particular when the number of predictors is large, there is a known risk of overfitting and poor predictive performance, since the effect of some predictors is constant over time. We propose a prior for variance shrinkage in TVP models, called triple gamma. The triple gamma prior encompasses a number of priors that have been suggested previously, such as the Bayesian lasso, the double gamma prior and the Horseshoe prior. We present the desirable p…
For analyzing unit-level multivariate data in small area estimation, we consider the multivariate nested error regression model (MNER) and provide the empirical best linear unbiased predictor (EBLUP) of a small area characteristic based on second-order unbiased and consistent estimators of the `within' and `between' multivariate components of variance. The second-order approximation of the mean squared error (MSE) matrix of the EBLUP and its unbiased estimator are derived in closed forms. The confidence interval with second-order accuracy is also provided analytically.
Beta regression model is useful in the analysis of bounded continuous outcomes such as proportions. It is well known that for any regression model, the presence of multicollinearity leads to poor performance of the maximum likelihood estimators. The ridge type estimators have been proposed to alleviate the adverse effects of the multicollinearity. Furthermore, when some of the predictors have insignificant or weak effects on the outcomes, it is desired to recover as much information as possible from these predictors instead of discarding them all together. In this paper we proposed ridge type …
This review traces the evolution of theory that started when Charles Stein in 1955 [In Proc. 3rd Berkeley Sympos. Math. Statist. Probab. I (1956) 197--206, Univ. California Press] showed that using each separate sample mean from $k\ge3$ Normal populations to estimate its own population mean $μ_i$ can be improved upon uniformly for every possible $μ=(μ_1,...,μ_k)'$. The dominating estimators, referred to here as being "Model-I minimax," can be found by shrinking the sample means toward any constant vector. Admissible minimax shrinkage estimators were derived by Stein and others as posterior mea…
In practice rarely (if ever) is the spatial covariance known in spatial prediction problems. Often, prediction is performed after estimated spatial covariance parameters are plugged into the prediction equation. The estimated spatial association parameters arealso plugged into the prediction variance of the spatial predictor. However, simply plugging in spatial covariance parameter estimates into the prediction variance of the spatial predictor does not take into account the uncertainty in the true values of the spatial covariance parameters. Therefore the plug-in prediction variance estimate …
Shrinkage estimates of small domain parameters typically utilize a combination of a noisy "direct" estimate that only uses data from a specific small domain and a more stable regression estimate. When the regression model is misspecified, estimation performance for the noisier domains can suffer due to substantial shrinkage towards a poorly estimated regression surface. In this paper, we introduce a new class of robust, empirically-driven regression weights that target estimation of the small domain means under potential misspecification of the global regression model. Our regression weights a…
Shrinkage estimation usually reduces variance at the cost of bias. But when we care only about some parameters of a model, I show that we can reduce variance without incurring bias if we have additional information about the distribution of covariates. In a linear regression model with homoscedastic Normal noise, I consider shrinkage estimation of the nuisance parameters associated with control variables. For at least three control variables and exogenous treatment, I establish that the standard least-squares estimator is dominated with respect to squared-error loss in the treatment effect eve…
Phase II basket trials are popular tools to evaluate efficacy of a new treatment targeting genetic alteration common to a set of different cancer histologies. Efficient designs are obtained by pooling data from the different arms (e.g., cancer histologies) via Bayesian hierarchical modelling, with a variance parameter controlling the strength of shrinkage of each arm treatment effect to the overall treatment effect. One critical aspect of this approach is that prior choice on the variance plays a major role in determining the strength of shrinkage and impacts the operating characteristics of t…
Ensemble Conditional Variance Estimation (ECVE) is a novel sufficient dimension reduction (SDR) method in regressions with continuous response and predictors. ECVE applies to general non-additive error regression models. It operates under the assumption that the predictors can be replaced by a lower dimensional projection without loss of information. It is a semiparametric forward regression model based exhaustive sufficient dimension reduction estimation method that is shown to be consistent under mild assumptions. It is shown to outperform central subspace mean average variance estimation (c…
To recover a low rank structure from a noisy matrix, truncated singular value decomposition has been extensively used and studied. Recent studies suggested that the signal can be better estimated by shrinking the singular values. We pursue this line of research and propose a new estimator offering a continuum of thresholding and shrinking functions. To avoid an unstable and costly cross-validation search, we propose new rules to select two thresholding and shrinking parameters from the data. In particular we propose a generalized Stein unbiased risk estimation criterion that does not require k…
High-dimensional data subject to heavy-tailed phenomena and heterogeneity are commonly encountered in various scientific fields and bring new challenges to the classical statistical methods. In this paper, we combine the asymmetric square loss and huber-type robust technique to develop the robust expectile regression for ultrahigh dimensional heavy-tailed heterogeneous data. Different from the classical huber method, we introduce two different tuning parameters on both sides to account for possibly asymmetry and allow them to diverge to reduce bias induced by the robust approximation. In the r…
Prediction of a vector of ordered parameters or part of it arises naturally in the context of Small Area Estimation (SAE). For example, one may want to estimate the parameters associated with the top ten areas, the best or worst area, or a certain percentile. We use a simple SAE model to show that estimation of ordered parameters by the corresponding ordered estimates of each area separately does not yield good results with respect to MSE. Shrinkage-type predictors, with an appropriate amount of shrinkage for the particular problem of ordered parameters, are considerably better, and their perf…
We express the mean and variance terms in a double exponential regression model as additive functions of the predictors and use Bayesian variable selection to determine which predictors enter the model, and whether they enter linearly or flexibly. When the variance term is null we obtain a generalized additive model, which becomes a generalized linear model if the predictors enter the mean linearly. The model is estimated using Markov chain Monte Carlo simulation and the methodology is illustrated using real and simulated data sets.
We consider how increasingly available observational data can be used to improve the design of randomized controlled trials (RCTs). We seek to design a prospective RCT, with the intent of using an Empirical Bayes estimator to shrink the causal estimates from our trial toward causal estimates obtained from an observational study. We ask: how might we design the experiment to better complement the observational study in this setting? We propose using an estimator that shrinks each component of the RCT causal estimator toward its observational counterpart by a factor proportional to its variance.…
This article considers a robust hierarchical Bayesian approach to deal with random effects of small area means when some of these effects assume extreme values, resulting in outliers. In presence of outliers, the standard Fay-Herriot model, used for modeling area-level data, under normality assumptions of the random effects may overestimate random effects variance, thus provides less than ideal shrinkage towards the synthetic regression predictions and inhibits borrowing information. Even a small number of substantive outliers of random effects result in a large estimate of the random effects …
For small area estimation of area-level data, the Fay-Herriot model is extensively used as a model based method. In the Fay-Herriot model, it is conventionally assumed that the sampling variances are known whereas estimators of sampling variances are used in practice. Thus, the settings of knowing sampling variances are unrealistic and several methods are proposed to overcome this problem. In this paper, we assume the situation where the direct estimators of the sampling variances are available as well as the sample means. Using these information, we propose a Bayesian yet objective method pro…
Selecting powerful predictors for an outcome is a cornerstone task for machine learning. However, some types of questions can only be answered by identifying the predictors that causally affect the outcome. A recent approach to this causal inference problem leverages the invariance property of a causal mechanism across differing experimental environments (Peters et al., 2016; Heinze-Deml et al., 2018). This method, invariant causal prediction (ICP), has a substantial computational defect -- the runtime scales exponentially with the number of possible causal variables. In this work, we show tha…
Conditional Variance Estimation (CVE) is a novel sufficient dimension reduction (SDR) method for additive error regressions with continuous predictors and link function. It operates under the assumption that the predictors can be replaced by a lower dimensional projection without loss of information. In contrast to the majority of moment based sufficient dimension reduction methods, Conditional Variance Estimation is fully data driven, does not require the restrictive linearity and constant variance conditions, and is not based on inverse regression. CVE is shown to be consistent and its objec…
This paper proposes a novel model-free screening procedure for ultrahigh dimensional data analysis. By utilizing slicing technique which has been successfully ap- plied to continuous variables, we construct a new index called the fused mean-variance for feature screening. This method has the following merits: (i) it is model-free, i.e., without specifying regression form of predictors and response variable; (ii) it can be used to analyze various types of variables including discrete, categorical and continuous vari- ables; (iii) it still works well even when the covariates/random errors are he…
One of the major challenges in multivariate analysis is the estimation of population covariance matrix from sample covariance matrix (SCM). Most recent covariance matrix estimators use either shrinkage transformations or asymptotic results from Random Matrix Theory (RMT). Shrinkage techniques help in pulling extreme correlation values towards certain target values whereas tools from RMT help in removing noisy eigenvalues of SCM. Both of these techniques use different approaches to achieve a similar goal which is to remove noisy correlations and add structure to SCM to overcome the bias-varianc…
