For the outlier problem in linear regression models, the Student-$t$ linear regression model is one of the common methods for robust modeling and is widely adopted in the literature. However, most of them applies it without careful theoretical consideration. This study provides the practically useful and quite simple conditions to ensure that the Student-$t$ linear regression model is robust against an outlier in the $y$-direction using regular variation theory.
This paper considers the Laplace method to derive approximate inference for the Gaussian process (GP) regression in the location and scale parameters of the Student-t probabilistic model. This allows both mean and variance of the data to vary as a function of covariates with the attractive feature that the Student-t model has been widely used as a useful tool for robustifying data analysis. The challenge in the approximate inference for the GP regression with the Student-t probabilistic model, lies in the analytical intractability of the posterior distribution and the lack of concavity of the …
Bayesian Student-$t$ linear regression is a common robust alternative to the normal model, but its theoretical properties are not well understood. We aim to fill some gaps by providing analyses in two different asymptotic scenarios. The results allow to precisely characterize the trade-off between robustness and efficiency controlled through the degrees of freedom (at least asymptotically).
This paper applies the minimum message length principle to inference of linear regression models with Student-t errors. A new criterion for variable selection and parameter estimation in Student-t regression is proposed. By exploiting properties of the regression model, we derive a suitable non-informative proper uniform prior distribution for the regression coefficients that leads to a simple and easy-to-apply criterion. Our proposed criterion does not require specification of hyperparameters and is invariant under both full rank transformations of the design matrix and linear transformations…
Large scale study of primary school student performance relative to their LMS activity and socioeconomic demographics using a Bayesian Additive Regression Trees containing random effects
Regression models with fat-tailed error terms are an increasingly popular choice to obtain more robust inference to the presence of outlying observations. This article focuses on Bayesian inference for the Student-$t$ linear regression model under objective priors that are based on the Jeffreys rule. Posterior propriety results presented in Fonseca et al. (2008) are revisited and corrected. In particular, it is shown that the standard Jeffreys-rule prior precludes the existence of a proper posterior distribution.
As an automatic method of determining model complexity using the training data alone, Bayesian linear regression provides us a principled way to select hyperparameters. But one often needs approximation inference if distribution assumption is beyond Gaussian distribution. In this paper, we propose a Bayesian linear regression model with Student-t assumptions (BLRS), which can be inferred exactly. In this framework, both conjugate prior and expectation maximization (EM) algorithm are generalized. Meanwhile, we prove that the maximum likelihood solution is equivalent to the standard Bayesian lin…
The choice of the prior distribution is a key aspect of Bayesian analysis. For the spatial regression setting a subjective prior choice for the parameters may not be trivial, from this perspective, using the objective Bayesian analysis framework a reference is introduced for the spatial Student-t regression model with unknown degrees of freedom. The spatial Student-t regression model poses two main challenges when eliciting priors: one for the spatial dependence parameter and the other one for the degrees of freedom. It is well-known that the propriety of the posterior distribution over object…
In logistic regression, separation occurs when a linear combination of the predictors can perfectly classify part or all of the observations in the sample, and as a result, finite maximum likelihood estimates of the regression coefficients do not exist. Gelman et al. (2008) recommended independent Cauchy distributions as default priors for the regression coefficients in logistic regression, even in the case of separation, and reported posterior modes in their analyses. As the mean does not exist for the Cauchy prior, a natural question is whether the posterior means of the regression coefficie…
Gaussian process model for vector-valued function has been shown to be useful for multi-output prediction. The existing method for this model is to re-formulate the matrix-variate Gaussian distribution as a multivariate normal distribution. Although it is effective in many cases, re-formulation is not always workable and is difficult to apply to other distributions because not all matrix-variate distributions can be transformed to respective multivariate distributions, such as the case for matrix-variate Student$-t$ distribution. In this paper, we propose a unified framework which is used not …
This work introduces a novel methodology based on finite mixtures of Student-t distributions to model the errors' distribution in linear regression models. The novelty lies on a particular hierarchical structure for the mixture distribution in which the first level models the number of modes, responsible to accommodate multimodality and skewness features, and the second level models tail behavior. Moreover, the latter is specified in a way that no degrees of freedom parameters are estimated and, therefore, the known statistical difficulties when dealing with those parameters is mitigated, and …
We consider the nonparametric multivariate isotonic regression problem, where the regression function is assumed to be nondecreasing with respect to each predictor. Our goal is to construct a Bayesian credible interval for the function value at a given interior point with assured limiting frequentist coverage. We put a prior on unrestricted step-functions, but make inference using the induced posterior measure by an "immersion map" from the space of unrestricted functions to that of multivariate monotone functions. This allows maintaining the natural conjugacy for posterior sampling. A natural…
Expected-posterior priors (EPP) have been proved to be extremely useful for testing hypothesis on the regression coefficients of normal linear models. One of the advantages of using EPPs is that impropriety of baseline priors causes no indeterminacy. However, in regression problems, they based on one or more \textit{training samples}, that could influence the resulting posterior distribution. The power-expected-posterior priors are minimally-informative priors that diminishing the effect of training samples on the EPP approach, by combining ideas from the power-prior and unit-information-prior…
We propose a new prior distribution for classical (nonhierarchical) logistic regression models, constructed by first scaling all nonbinary variables to have mean 0 and standard deviation 0.5, and then placing independent Student-$t$ prior distributions on the coefficients. As a default choice, we recommend the Cauchy distribution with center 0 and scale 2.5, which in the simplest setting is a longer-tailed version of the distribution attained by assuming one-half additional success and one-half additional failure in a logistic regression. Cross-validation on a corpus of datasets shows the Cauc…
The problem motivating the paper is the quantification of students' preferences regarding teaching/coursework quality, under certain numerical restrictions, in order to build a model for identifying, assessing and monitoring the major components of the overall academic quality. After reviewing the strengths and limitations of conjoint analysis and of the random coefficient regression model used in similar problems in the past, we propose a Bayesian beta regression model with a Dirichlet prior on the model coefficients. This approach not only allows for the incorporation of informative prior wh…
SigOpt Inc.（SigOpt公司）；Centro Universitario de la Defensa, Zaragoza（萨拉戈萨国防大学中心）
Bayesian optimization has recently attracted the attention of the automatic machine learning community for its excellent results in hyperparameter tuning. BO is characterized by the sample efficiency with which it can optimize expensive black-box functions. The efficiency is achieved in a similar fashion to the learning to learn methods: surrogate models (typically in the form of Gaussian processes) learn the target function and perform intelligent sampling. This surrogate model can be applied even in the presence of noise; however, as with most regression methods, it is very sensitive to outl…
The Student-$t$ distribution is widely used in statistical modeling of datasets involving outliers since its longer-than-normal tails provide a robust approach to hand such data. Furthermore, data collected over time may contain censored or missing observations, making it impossible to use standard statistical procedures. This paper proposes an algorithm to estimate the parameters of a censored linear regression model when the regression errors are autocorrelated and the innovations follow a Student-$t$ distribution. To fit the proposed model, maximum likelihood estimates are obtained througho…
This paper considers the robust and efficient implementation of Gaussian process regression with a Student-t observation model. The challenge with the Student-t model is the analytically intractable inference which is why several approximative methods have been proposed. The expectation propagation (EP) has been found to be a very accurate method in many empirical studies but the convergence of the EP is known to be problematic with models containing non-log-concave site functions such as the Student-t distribution. In this paper we illustrate the situations where the standard EP fails to conv…
This paper presents a Bayesian approach to multiple-output quantile regression. The unconditional model is proven to be consistent and asymptotically correct frequentist confidence intervals can be obtained. The prior for the unconditional model can be elicited as the ex-ante knowledge of the distance of the tau-Tukey depth contour to the Tukey median, the first prior of its kind. A proposal for conditional regression is also presented. The model is applied to the Tennessee Project Steps to Achieving Resilience (STAR) experiment and it finds a joint increase in tau-quantile subpopulations for …
We introduce flexible robust functional regression models, using various heavy-tailed processes, including a Student $t$-process. We propose efficient algorithms in estimating parameters for the marginal mean inferences and in predicting conditional means as well interpolation and extrapolation for the subject-specific inferences. We develop bootstrap prediction intervals for conditional mean curves. Numerical studies show that the proposed model provides robust analysis against data contamination or distribution misspecification, and the proposed prediction intervals maintain the nominal conf…
Consider the multiple linear regression model $y_{i} = \boldsymbol{x}'_{i} \boldsymbolβ + ε_{i}$, where $ε_i$'s are independent and identically distributed random variables, $\mathbf{x}_i$'s are known design vectors and $\boldsymbolβ$ is the $p \times 1$ vector of parameters. An effective way of approximating the distribution of the M-estimator $\boldsymbol{\barβ}_n$, after proper centering and scaling, is the Perturbation Bootstrap Method. In this current work, second order results of this non-naive bootstrap method have been investigated. Second order correctness is important for reducing th…
This paper examines a novel gradient boosting framework for regression. We regularize gradient boosted trees by introducing subsampling and employ a modified shrinkage algorithm so that at every boosting stage the estimate is given by an average of trees. The resulting algorithm, titled Boulevard, is shown to converge as the number of trees grows. We also demonstrate a central limit theorem for this limit, allowing a characterization of uncertainty for predictions. A simulation study and real world examples provide support for both the predictive accuracy of the model and its limiting behavior…
