A new generalized ridge regression shrinkage path is proposed that is as short as possible under the restriction that it must pass through the vector of regression coefficient estimators that make the overall Optimal Variance-Bias Trade-Off under Normal distribution-theory. Five distinct types of ridge TRACE displays plus other graphics for this efficient path are motivated and illustrated here. These visualizations provide invaluable data-analytic insights and improved self-confidence to researchers and data scientists fitting linear models to ill-conditioned (confounded) data.
When developing risk prediction models, shrinkage methods are recommended, especially when the sample size is limited. Several earlier studies have shown that the shrinkage of model coefficients can reduce overfitting of the prediction model and subsequently result in better predictive performance on average. In this simulation study, we aimed to investigate the variability of regression shrinkage on predictive performance for a binary outcome, with focus on the calibration slope. The slope indicates whether risk predictions are too extreme (slope < 1) or not extreme enough (slope > 1). We inv…
Penalization of the likelihood by Jeffreys' invariant prior, or by a positive power thereof, is shown to produce finite-valued maximum penalized likelihood estimates in a broad class of binomial generalized linear models. The class of models includes logistic regression, where the Jeffreys-prior penalty is known additionally to reduce the asymptotic bias of the maximum likelihood estimator; and also models with other commonly used link functions such as probit and log-log. Shrinkage towards equiprobability across observations, relative to the maximum likelihood estimator, is established theore…
In this note, we explore a simple approach to composition estimation, using penalized likelihood density estimation on a nominal discrete domain. Practical issues such as smoothing parameter selection and the use of prior information are investigated in simulations, and a theoretical analysis is attempted. The method has been implemented in a pair of R functions for use by practitioners.
In this paper, we present a method of maximum a posteriori estimation of parameters in dynamic factor models with incomplete data. We extend maximum likelihood expectation maximization iterations by Bańbura & Modugno (2014) to penalized counterparts by applying parameter shrinkage in a Minnesota prior style fashion, also considering factors loading onto variables dynamically. A heuristic and adapting shrinkage scheme is considered. The algorithm is applicable to any arbitrary pattern of missing data, including different publication dates, sample lengths and frequencies. The method is evaluated…
Separation in logistic regression is a common problem causing failure of the iterative estimation process when finding maximum likelihood estimates. Firth's correction (FC) was proposed as a solution, providing estimates also in presence of separation. In this paper we evaluate whether ridge regression (RR) could be considered instead, specifically, if it could reduce the mean squared error (MSE) of coefficient estimates in comparison to FC. In RR the tuning parameter determining the penalty strength is usually obtained by minimizing some measure of the out-of-sample prediction error or inform…
In many applications (e.g., medical studies), the population of interest (e.g., disease status) comprises heterogeneous subpopulations. The mixture of probabilistic regression models is one of the most common techniques to incorporate the information of covariates into learning of the population heterogeneity. Despite its flexibility, the model may lead to unreliable estimates in the presence of multicollinearity problem. In this paper, we develop Liu-type shrinkage methods through an unsupervised learning approach to estimate the model coefficients in multicollinearity. The performance of the…
This paper addresses the task of estimating a covariance matrix under a patternless sparsity assumption. In contrast to existing approaches based on thresholding or shrinkage penalties, we propose a likelihood-based method that regularizes the distance from the covariance estimate to a symmetric sparsity set. This formulation avoids unwanted shrinkage induced by more common norm penalties and enables optimization of the resulting non-convex objective by solving a sequence of smooth, unconstrained subproblems. These subproblems are generated and solved via the proximal distance version of the m…
My first paper exclusively about ridge regression was published in Technometrics and chosen for invited presentation at the 1975 Joint Statistical Meetings in Atlanta. Unfortunately, that paper contained a wide range of assorted details and results. Luckily, Gary McDonald's published discussion of that paper focused primarily on my use of Maximum Likelihood estimation under normal distribution-theory. In this review of some results from all four of my ridge publications between 1975 and 2022, I highlight the Maximum Likelihood findings that appear to be most important in practical application …
Longitudinal item response data are common in social science, educational science, and psychology, among other disciplines. Studying the time-varying relationships between items is crucial for educational assessment or designing marketing strategies from survey questions. Although dynamic network models have been widely developed, we cannot apply them directly to item response data because there are multiple systems of nodes with various types of local interactions among items, resulting in multiplex network structures. We propose a new model to study these temporal interactions among items by…
The estimation of covariance matrices of multiple classes with limited training data is a difficult problem. The sample covariance matrix (SCM) is known to perform poorly when the number of variables is large compared to the available number of samples. In order to reduce the mean squared error (MSE) of the SCM, regularized (shrinkage) SCM estimators are often used. In this work, we consider regularized SCM (RSCM) estimators for multiclass problems that couple together two different target matrices for regularization: the pooled (average) SCM of the classes and the scaled identity matrix. Regu…
In all areas of human knowledge, datasets are increasing in both size and complexity, creating the need for richer statistical models. This trend is also true for economic data, where high-dimensional and nonlinear/nonparametric inference is the norm in several fields of applied econometric work. The purpose of this paper is to introduce the reader to the world of Bayesian model determination, by surveying modern shrinkage and variable selection algorithms and methodologies. Bayesian inference is a natural probabilistic framework for quantifying uncertainty and learning about model parameters,…
A separable covariance model for a random matrix provides a parsimonious description of the covariances among the rows and among the columns of the matrix, and permits likelihood-based inference with a very small sample size. However, in many applications the assumption of exact separability is unlikely to be met, and data analysis with a separable model may overlook or misrepresent important dependence patterns in the data. In this article, we propose a compromise between separable and unstructured covariance estimation. We show how the set of covariance matrices may be uniquely parametrized …
This paper studies the E-Bayesian (expectation of the Bayesian estimation) estimation of the parameter of Lomax distribution based on different loss functions. Under different loss functions, we calculate the Bayesian estimation of the parameter and then calculate the expectation of the estimated value to get the E-Bayesian estimation. To measure the estimated error, the E-MSE (expected mean squared error) is introduced. And the formulas of E-Bayesian estimation and E-MSE are given. By applying Markov Chain Monte Carlo technology, we analyze the performances of the proposed methods. Results ar…
Beta regression model is useful in the analysis of bounded continuous outcomes such as proportions. It is well known that for any regression model, the presence of multicollinearity leads to poor performance of the maximum likelihood estimators. The ridge type estimators have been proposed to alleviate the adverse effects of the multicollinearity. Furthermore, when some of the predictors have insignificant or weak effects on the outcomes, it is desired to recover as much information as possible from these predictors instead of discarding them all together. In this paper we proposed ridge type …
Linear discriminant analysis (LDA) is a typical method for classification problems with large dimensions and small samples. There are various types of LDA methods that are based on the different types of estimators for the covariance matrices and mean vectors. In this paper, we consider shrinkage methods based on a non-parametric approach. For the precision matrix, methods based on the sparsity structure or data splitting are examined. Regarding the estimation of mean vectors, Non-parametric Empirical Bayes (NPEB) methods and Non-parametric Maximum Likelihood Estimation (NPMLE) methods, also k…
For finite samples with binary outcomes penalized logistic regression such as ridge logistic regression (RR) has the potential of achieving smaller mean squared errors (MSE) of coefficients and predictions than maximum likelihood estimation. There is evidence, however, that RR is sensitive to small or sparse data situations, yielding poor performance in individual datasets. In this paper, we elaborate this issue further by performing a comprehensive simulation study, investigating the performance of RR in comparison to Firth's correction that has been shown to perform well in low-dimensional s…
In linear regression, the least squares (LS) estimator has certain optimality properties if the errors are normally distributed. This assumption is often violated in practice, partly caused by data outliers. Robust estimators can cope with this situation and thus they are widely used in practice. One example of robust estimators for regression are adaptive modified maximum likelihood (AMML) estimators (Donmez, 2010). However, they are not robust to $x$ outliers, so-called leverage points. In this study, we propose a new regression estimator by employing an appropriate weighting scheme in the A…
Linear regression models are useful statistical tools to analyze data sets in several different fields. There are several methods to estimate the parameters of a linear regression model. These methods usually perform under normally distributed and uncorrelated errors with zero mean and constant variance. However, for some data sets error terms may not satisfy these or some of these assumptions. If error terms are correlated, such as the regression models with autoregressive (AR(p)) error terms, the Conditional Maximum Likelihood (CML) under normality assumption or the Least Square (LS) methods…
Most of previous works and applications of Bayesian factor model have assumed the normal likelihood regardless of its validity. We propose a Bayesian factor model for heavy-tailed high-dimensional data based on multivariate Student-$t$ likelihood to obtain better covariance estimation. We use multiplicative gamma process shrinkage prior and factor number adaptation scheme proposed in Bhattacharya & Dunson [Biometrika (2011) 291-306]. Since a naive Gibbs sampler for the proposed model suffers from slow mixing, we propose a Markov Chain Monte Carlo algorithm where fast mixing of Hamiltonian Mont…
