We consider the Bayesian binary regression model and we introduce a new class of distributions, the Perturbed Unified Skew-Normal (pSUN, henceforth), which generalizes the Unified Skew-Normal (SUN) class. We show that the new class is conjugate to any binary regression model, provided that the link function may be expressed as a scale mixture of Gaussian CDFs. We discuss in detail the popular logit case, and we show that, when a logistic regression model is combined with a Gaussian prior, posterior summaries such as cumulants and normalizing constants can easily be obtained through the use of an importance sampling approach, opening the way to straightforward variable selection procedures. For more general prior distributions, the proposed methodology is based on a simple Gibbs sampler algorithm. We also claim that, in the p>n case, our proposal presents better performances - both in terms of mixing and accuracy - compared to the existing methods.
Multivariate stochastic volatility models with skew distributions are proposed. Exploiting Cholesky stochastic volatility modeling, univariate stochastic volatility processes with leverage effect and generalized hyperbolic skew t-distributions are embedded to multivariate analysis with time-varying correlations. Bayesian prior works allow this approach to provide parsimonious skew structure and to easily scale up for high-dimensional problem. Analyses of daily stock returns are illustrated. Empirical results show that the time-varying correlations and the sparse skew structure contribute to im…
Harvey et al. (2010) extended the Bayesian estimation method by Sahu et al. (2003) to a multivariate skew-elliptical distribution with a general skewness matrix, and applied it to Bayesian portfolio optimization with higher moments. Although their method is epochal in the sense that it can handle the skewness dependency among asset returns and incorporate higher moments into portfolio optimization, it cannot identify all elements in the skewness matrix due to label switching in the Gibbs sampler. To deal with this identification issue, we propose to modify their sampling algorithm by imposing …
Insurance risks data typically exhibit skewed behaviour. In this paper, we propose a Bayesian approach to capture the main features of these datasets. This work extends the methodology introduced in Villa and Walker (2014a) by considering an extra parameter which captures the skewness of the data. In particular, a skewed Student-t distribution is considered. Two datasets are analysed: the Danish fire losses and the US indemnity loss. The analysis is carried with an objective Bayesian approach. For the discrete parameter representing the number of the degrees of freedom, we adopt a novel prior …
A brief comment on A variational Bayesian approach for inverse problems with skew-t error distributions (Guha et al., Journal of Computational Physics 301 (2015) 377-393) is given in this letter.
We perform a Bayesian analysis of the p-variate skew-t model, providing a new parameterization, a set of non-informative priors and a sampler specifically designed to explore the posterior density of the model parameters. Extensions, such as the multivariate regression model with skewed errors and the stochastic frontiers model, are easily accommodated. A novelty introduced in the paper is given by the extension of the bivariate skew-normal model given in Liseo & Parisi (2013) to a more realistic p-variate skew-t model. We also introduce the R package mvst, which allows to estimate the multiva…
Trinity College（都柏林圣三一学院）；Dalle Molle Institute for Artificial Intelligence Research (IDSIA)（达尔莫尔人工智能研究所（IDSIA））
Skew-Gaussian processes (SkewGPs) extend the multivariate Unified Skew-Normal distributions over finite dimensional vectors to distribution over functions. SkewGPs are more general and flexible than Gaussian processes, as SkewGPs may also represent asymmetric distributions. In a recent contribution we showed that SkewGP and probit likelihood are conjugate, which allows us to compute the exact posterior for non-parametric binary classification and preference learning. In this paper, we generalize previous results and we prove that SkewGP is conjugate with both the normal and affine probit likel…
Skewed generalizations of the normal distribution have been a topic of great interest in the statistics community due to their diverse applications across several domains. One of the most popular skew normal distributions, due to its intuitive appeal, is the Azzalini's skew normal distribution. However, due to the nature of the distribution it suffers from serious inferential problems. Interestingly, the Bayesian approach has been shown to mitigate these issues. Recently, another skew normal distribution, the Geometric skew normal distribution, which is structurally different from Azzalini's s…
In this note we introduce linear regression with basis functions in order to apply Bayesian model selection. The goal is to incorporate Occam's razor as provided by Bayes analysis in order to automatically pick the model optimally able to explain the data without overfitting.
This paper describes a new Bayesian interpretation of a class of skew--Student $t$ distributions. We consider a hierarchical normal model with unknown covariance matrix and show that by imposing different restrictions on the parameter space, corresponding Bayes predictive density estimators under Kullback-Leibler loss function embrace some well-known skew--Student $t$ distributions. We show that obtained estimators perform better in terms of frequentist risk function over regular Bayes predictive density estimators. We apply our proposed methods to estimate future densities of medical data: th…
The multivariate extended skew-normal distribution allows for accommodating raw data which are skewed and heavy tailed, and has at least three appealing statistical properties, namely closure under conditioning, affine transformations, and marginalization. In this paper we propose a Bayesian computational approach based on a sequential Monte Carlo (SMC) sampler to estimate such distributions. The practical implementation of each step of the algorithm is discussed and the elicitation of prior distributions takes into consideration some unusual behaviour of the likelihood function and the corres…
We present a proposal to deal with the non-normality issue in the context of regression models with measurement errors when both the response and the explanatory variable are observed with error. We extend the normal model by jointly modeling the unobserved covariate and the random errors by a finite mixture of scale mixture of skew-normal distributions. This approach allows us to model data with great flexibility, accommodating skewness, heavy tails, and multi-modality.
Bayesian Student-$t$ linear regression is a common robust alternative to the normal model, but its theoretical properties are not well understood. We aim to fill some gaps by providing analyses in two different asymptotic scenarios. The results allow to precisely characterize the trade-off between robustness and efficiency controlled through the degrees of freedom (at least asymptotically).
In this paper, we present a Weibull link (skewed) model for categorical response data arising from binomial as well as multinomial model. We show that, for such types of categorical data, the most commonly used models (logit, probit and complementary log-log) can be obtained as limiting cases. We further compare the proposed model with some other asymmetrical models. The Bayesian as well as frequentist estimation procedures for binomial and multinomial data responses are presented in details. The analysis of two data sets to show the efficiency of the proposed model is performed.
Frequentist and likelihood methods of inference based on the multivariate skew-normal model encounter several technical difficulties with this model. In spite of the popularity of this class of densities, there are no broadly satisfactory solutions for estimation and testing problems. A general population Monte Carlo algorithm is proposed which: 1) exploits the latent structure stochastic representation of skew-normal random variables to provide a full Bayesian analysis of the model and 2) accounts for the presence of constraints in the parameter space. The proposed approach can be defined as …
Motivated by the analysis of the distribution of university grades, which is usually asymmetric, we discuss two informative priors for the shape parameter of the skew-normal distribution, showing that they lead to closed-form full-conditional posterior distributions, particularly useful in MCMC computation. Gibbs sampling algorithms are discussed for the joint vector of parameters, given independent prior distributions for the location and scale parameters. Simulation studies are performed to assess the performance of Gibbs samplers and to compare the choice of informative priors against a non…
We consider Bayesian inference in inverse regression problems where the objective is to infer about unobserved covariates from observed responses and covariates. We establish posterior consistency of such unobserved covariates in Bayesian inverse regression problemsunder appropriate priors in a leave-one-out cross-validation setup. We relate this to posterior consistency of inverse reference distributions (Bhattacharya (2013)) for assessing model adequacy. We illustrate our theory and methods with various examples of Bayesian inverse regression, along with adequate simulation experiments.
Regression models for dichotomous data are ubiquitous in statistics. Besides being useful for inference on binary responses, these methods serve also as building blocks in more complex formulations, such as density regression, nonparametric classification and graphical models. Within the Bayesian framework, inference proceeds by updating the priors for the coefficients, typically set to be Gaussians, with the likelihood induced by probit or logit regressions for the responses. In this updating, the apparent absence of a tractable posterior has motivated a variety of computational methods, incl…
In this paper, we suggest using a skew Gaussian-log Gaussian model for the analysis of spatial censored data from a Bayesian point of view. This approach furnishes an extension of the skew log Gaussian model to accommodate to both skewness and heavy tails and also censored data. All of the characteristics mentioned are three pervasive features of spatial data. We utilize data augmentation method and Markov chain Monte Carlo (MCMC) algorithms to do posterior calculations. The methodology is illustrated using simulated data, as well as applying it to a real data set. Keywords: Censored data, dat…
The asymmetric Laplace density (ALD) is used as a working likelihood for Bayesian quantile regression. Sriram et al.(2013) derived posterior consistency for Bayesian linear quantile regression based on the misspecified ALD. While their paper also argued for $\sqrt{n}-$consistency, Sriram and Ramamoorthi (2017) highlighted that the argument was only valid for a rate less than $\sqrt{n}$. So, the question of $\sqrt{n}-$rate has remained unaddressed, but is necessary to carry out meaningful Bayesian inference based on the ALD. In this paper, we derive results to obtain posterior consistency rates…
We formalise and generalise the definition of the family of univariate double two--piece distributions, obtained by using a density--based transformation of unimodal symmetric continuous distributions with a shape parameter. The resulting distributions contain five interpretable parameters that control the mode, as well as the scale and shape in each direction. Four-parameter subfamilies of this class of distributions that capture different types of asymmetry are discussed. We propose interpretable scale and location-invariant benchmark priors and derive conditions for the propriety of the cor…
Many machine learning tools for regression are based on recursive partitioning of the covariate space into smaller regions, where the regression function can be estimated locally. Among these, regression trees and their ensembles have demonstrated impressive empirical performance. In this work, we shed light on the machinery behind Bayesian variants of these methods. In particular, we study Bayesian regression histograms, such as Bayesian dyadic trees, in the simple regression case with just one predictor. We focus on the reconstruction of regression surfaces that are piecewise constant, where…
This paper is focused on dimension-free PAC-Bayesian bounds, under weak polynomial moment assumptions, allowing for heavy tailed sample distributions. It covers the estimation of the mean of a vector or a matrix, with applications to least squares linear regression. Special efforts are devoted to the estimation of Gram matrices, due to their prominent role in high-dimension data analysis.
Most Item Response Theory (IRT) models for dichotomous responses are based on probit or logit link functions which assume a symmetric relationship between the probability of a correct response and the latent traits of individuals submitted to a test. This assumption restricts the use of those models to the case in which all items have a symmetric behaviour. On the other hand, asymmetric models proposed in the literature impose that all the items in a test have an asymmetric behaviour. This assumption is inappropriate for great part of the tests which are, in general, composed by both symmetric…
We consider three Bayesian penalized regression models and show that the respective deterministic scan Gibbs samplers are geometrically ergodic regardless of the dimension of the regression problem. We prove geometric ergodicity of the Gibbs samplers for the Bayesian fused lasso, the Bayesian group lasso, and the Bayesian sparse group lasso. Geometric ergodicity along with a moment condition results in the existence of a Markov chain central limit theorem for Monte Carlo averages and ensures reliable output analysis. Our results of geometric ergodicity allow us to also provide default starting…
Leiden University（莱顿大学）；CWI（荷兰国家数学与计算机科学研究所）；University of Oxford（牛津大学）；University of Victoria（维多利亚大学）
We study generalized Bayesian inference under misspecification, i.e. when the model is 'wrong but useful'. Generalized Bayes equips the likelihood with a learning rate $η$. We show that for generalized linear models (GLMs), $η$-generalized Bayes concentrates around the best approximation of the truth within the model for specific $η\neq 1$, even under severely misspecified noise, as long as the tails of the true distribution are exponential. We derive MCMC samplers for generalized Bayesian lasso and logistic regression and give examples of both simulated and real-world data in which generalize…
Non-Gaussian mixture models are gaining increasing attention for mixture model-based clustering particularly when dealing with data that exhibit features such as skewness and heavy tails. Here, such a mixture distribution is presented, based on the multivariate normal inverse Gaussian (MNIG) distribution. For parameter estimation of the mixture, a Bayesian approach via Gibbs sampler is used; for this, a novel approach to simulate univariate generalized inverse Gaussian random variables and matrix generalized inverse Gaussian random matrices is provided. The proposed algorithm will be applied t…
We consider Bayesian shrinkage predictions for the Normal regression problem under the frequentist Kullback-Leibler risk function. Firstly, we consider the multivariate Normal model with an unknown mean and a known covariance. While the unknown mean is fixed, the covariance of future samples can be different from training samples. We show that the Bayesian predictive distribution based on the uniform prior is dominated by that based on a class of priors if the prior distributions for the covariance and future covariance matrices are rotation invariant. Then, we consider a class of priors for t…
In this paper we deal with the regression problem in a random design setting. We investigate asymptotic optimality under minimax point of view of various Bayesian rules based on warped wavelets and show that they nearly attain optimal minimax rates of convergence over the Besov smoothness class considered. Warped wavelets have been introduced recently, they offer very good computable and easy-to-implement properties while being well adapted to the statistical problem at hand. We particularly put emphasis on Bayesian rules leaning on small and large variance Gaussian priors and discuss their si…
We develop a unifying framework for Bayesian nonparametric regression to study the rates of contraction with respect to the integrated $L_2$-distance without assuming the regression function space to be uniformly bounded. The framework is very flexible and can be applied to a wide class of nonparametric prior models. Three non-trivial applications of the proposed framework are provided: The finite random series regression of an $α$-Hölder function, with adaptive rates of contraction up to a logarithmic factor; The un-modified block prior regression of an $α$-Sobolev function, with adaptive-and…
We study frequentist asymptotic properties of Bayesian procedures for high-dimensional Gaussian sparse regression when unknown nuisance parameters are involved. Nuisance parameters can be finite-, high-, or infinite-dimensional. A mixture of point masses at zero and continuous distributions is used for the prior distribution on sparse regression coefficients, and appropriate prior distributions are used for nuisance parameters. The optimal posterior contraction of sparse regression coefficients, hampered by the presence of nuisance parameters, is also examined and discussed. It is shown that t…
The paper deals with the non-parametric estimation in the regression with the multiplicative noise. Using the local polynomial fitting and the bayesian approach, we construct the minimax on isotropic Hölder class estimator. Next applying Lepski's method, we propose the estimator which is optimally adaptive over the collection of isotropic Hölder classes. To prove the optimality of the proposed procedure we establish, in particular, the exponential inequality for the deviation of locally bayesian estimator from the parameter to be estimated. These theoretical results are illustrated by simulati…
Since their inception in the 1980's, regression trees have been one of the more widely used non-parametric prediction methods. Tree-structured methods yield a histogram reconstruction of the regression surface, where the bins correspond to terminal nodes of recursive partitioning. Trees are powerful, yet susceptible to over-fitting. Strategies against overfitting have traditionally relied on pruning greedily grown trees. The Bayesian framework offers an alternative remedy against overfitting through priors. Roughly speaking, a good prior charges smaller trees where overfitting does not occur. …
In the problem of selecting variables in a multivariate linear regression model, we derive new Bayesian information criteria based on a prior mixing a smooth distribution and a delta distribution. Each of them can be interpreted as a fusion of the Akaike information criterion (AIC) and the Bayesian information criterion (BIC). Inheriting their asymptotic properties, our information criteria are consistent in variable selection in both the large-sample and the high-dimensional asymptotic frameworks. In numerical simulations, variable selection methods based on our information criteria choose th…
The logistic regression model is the most popular model for analyzing binary data. In the absence of any prior information, an improper flat prior is often used for the regression coefficients in Bayesian logistic regression models. The resulting intractable posterior density can be explored by running Polson et al.'s (2013) data augmentation (DA) algorithm. In this paper, we establish that the Markov chain underlying Polson et al.'s (2013) DA algorithm is geometrically ergodic. Proving this theoretical result is practically important as it ensures the existence of central limit theorems (CLTs…
We investigate two empirical Bayes methods and a hierarchical Bayes method for adapting the scale of a Gaussian process prior in a nonparametric regression model. We show that all methods lead to a posterior contraction rate that adapts to the smoothness of the true regression function. Furthermore, we show that the corresponding credible sets cover the true regression function whenever this function satisfies a certain extrapolation condition. This condition depends on the specific method, but is implied by a condition of self-similarity. The latter condition is shown to be satisfied with pro…
We propose a new, two-step empirical Bayes-type of approach for neural networks. We show in context of the nonparametric regression model that the procedure (up to a logarithmic factor) provides optimal recovery of the underlying functional parameter of interest and provides Bayesian credible sets with frequentist coverage guarantees. The approach requires fitting the neural network only once, hence it is substantially faster than Bootstrapping type approaches. We demonstrate the applicability of our method over synthetic data, observing good estimation properties and reliable uncertainty quan…
During the past decade, shrinkage priors have received much attention in Bayesian analysis of high-dimensional data. This paper establishes the posterior consistency for high-dimensional linear regression with a class of shrinkage priors, which has a heavy and flat tail and allocates a sufficiently large probability mass in a very small neighborhood of zero. While enjoying its efficiency in posterior simulations, the shrinkage prior can lead to a nearly optimal posterior contraction rate and variable selection consistency as the spike-and-slab prior. Our numerical results show that under the p…
In this paper, we discuss the method of Bayesian regression and its efficacy for predicting price variation of Bitcoin, a recently popularized virtual, cryptographic currency. Bayesian regression refers to utilizing empirical data as proxy to perform Bayesian inference. We utilize Bayesian regression for the so-called "latent source model". The Bayesian regression for "latent source model" was introduced and discussed by Chen, Nikolov and Shah (2013) and Bresler, Chen and Shah (2014) for the purpose of binary classification. They established theoretical as well as empirical efficacy of the met…
This work affords new insights into Bayesian CART in the context of structured wavelet shrinkage. The main thrust is to develop a formal inferential framework for Bayesian tree-based regression. We reframe Bayesian CART as a g-type prior which departs from the typical wavelet product priors by harnessing correlation induced by the tree topology. The practically used Bayesian CART priors are shown to attain adaptive near rate-minimax posterior concentration in the supremum norm in regression models. For the fundamental goal of uncertainty quantification, we construct adaptive confidence bands f…
A two-stage approach is proposed to overcome the problem in quantile regression, where separately fitted curves for several quantiles may cross. The standard Bayesian quantile regression model is applied in the first stage, followed by a Gaussian process regression adjustment, which monotonizes the quantile function whilst borrowing strength from nearby quantiles. The two stage approach is computationally efficient, and more general than existing techniques. The method is shown to be competitive with alternative approaches via its performance in simulated examples.
In this paper we extend existing Bayesian methods for variable selection in Gaussian process regression, to select both the regression terms and the active covariates in the spatial correlation structure. We then use the estimated posterior probabilities to choose between relatively few modes through cross-validation, and consequently improve prediction.
This paper proposes factor stochastic volatility models with skew error distributions. The generalized hyperbolic skew t-distribution is employed for common-factor processes and idiosyncratic shocks. Using a Bayesian sparsity modeling strategy for the skewness parameter provides a parsimonious skew structure for possibly high-dimensional stochastic volatility models. Analyses of daily stock returns are provided. Empirical results show that the skewness is important for common-factor processes but less for idiosyncratic shocks. The sparse skew structure improves prediction and portfolio perform…
Trinity College（都柏林圣三一学院）；Dalle Molle Institute for Artificial Intelligence Research (IDSIA) - USI/SUPSI（达勒莫尔人工智能研究所（IDSIA）- 瑞士意大利语区应用艺术与科学大学/瑞士南部应用科学与艺术大学）
Preferential Bayesian optimisation (PBO) deals with optimisation problems where the objective function can only be accessed via preference judgments, such as "this is better than that" between two candidate solutions (like in A/B tests or recommender systems). The state-of-the-art approach to PBO uses a Gaussian process to model the preference function and a Bernoulli likelihood to model the observed pairwise comparisons. Laplace's method is then employed to compute posterior inferences and, in particular, to build an appropriate acquisition function. In this paper, we prove that the true post…
This study aims to show the fundamental difference between logistic regression and Bayesian classifiers in the case of exponential and unexponential families of distributions, yielding the following findings. First, the logistic regression is a less general representation of a Bayesian classifier. Second, one should suppose distributions of classes for the correct specification of logistic regression equations. Third, in specific cases, there is no difference between predicted probabilities from correctly specified generative Bayesian classifier and discriminative logistic regression.
We study Bayesian linear regression models with skew-symmetric scale mixtures of normal error distributions. These kinds of models can be used to capture departures from the usual assumption of normality of the errors in terms of heavy tails and asymmetry. We propose a general non-informative prior structure for these regression models and show that the corresponding posterior distribution is proper under mild conditions. We extend these propriety results to cases where the response variables are censored. The latter scenario is of interest in the context of accelerated failure time models, wh…
These notes aim at clarifying different strategies to perform linear regression from given dataset. Methods like the weighted and ordinary least squares, ridge regression or LASSO are proposed in the literature. The present article is my understanding of these methods which are, according to me, better unified in the Bayesian framework. The formulas to address linear regression with these methods are derived. The KIC for model selection is also derived in the end of the document.
We derive an exact and efficient Bayesian regression algorithm for piecewise constant functions of unknown segment number, boundary location, and levels. It works for any noise and segment level prior, e.g. Cauchy which can handle outliers. We derive simple but good estimates for the in-segment variance. We also propose a Bayesian regression curve as a better way of smoothing data without blurring boundaries. The Bayesian approach also allows straightforward determination of the evidence, break probabilities and error estimates, useful for model selection and significance and robustness studie…
Regression for spatially dependent outcomes poses many challenges, for inference and for computation. Non-spatial models and traditional spatial mixed-effects models each have their advantages and disadvantages, making it difficult for practitioners to determine how to carry out a spatial regression analysis. We discuss the data-generating mechanisms implicitly assumed by various popular spatial regression models, and discuss the implications of these assumptions. We propose Bayesian spatial filtering as an approximate middle way between non-spatial models and traditional spatial mixed models.…
Although linear regression models are fundamental tools in statistical science, the estimation results can be sensitive to outliers. While several robust methods have been proposed in frequentist frameworks, statistical inference is not necessarily straightforward. We here propose a Bayesian approach to robust inference on linear regression models using synthetic posterior distributions based on $γ$-divergence, which enables us to naturally assess the uncertainty of the estimation through the posterior distribution. We also consider the use of shrinkage priors for the regression coefficients t…
Recently, a very attractive logistic regression inference method for exponential family Gibbs spatial point processes was introduced. We combined it with the technique of quadratic tangential variational approximation and derived a new Bayesian technique for analysing spatial point patterns. The technique is described in detail, and demonstrated on numerical examples.
