We consider the Jeffreys-Lindley paradox from an objective Bayesian perspective by attempting to find priors representing complete indifference to sample size in the problem. This means that we ensure that the prior for the unknown mean and the prior predictive for the $t$-statistic are independent of the sample size. If successful, this would lead to Bayesian model comparison that was independent of sample size and ameliorate the paradox. Unfortunately, it leads to an improper scale-invariant prior for the unknown mean. We show, however, that a truncated scale-invariant prior delays the dependence on sample size, which could be practically significant. Lastly, we shed light on the paradox by relating it to the fact that the scale-invariant prior is improper.
This paper discusses the dual interpretation of the Jeffreys--Lindley's paradox associated with Bayesian posterior probabilities and Bayes factors, both as a differentiation between frequentist and Bayesian statistics and as a pointer to the difficulty of using improper priors while testing. We stress the considerable impact of this paradox on the foundations of both classical and Bayesian statistics. While assessing existing resolutions of the paradox, we focus on a critical viewpoint of the paradox discussed by Spanos (2013) in Philosophy of Science.
This paper is concerned with the well known Jeffreys-Lindley paradox. In a Bayesian set up, the so-called paradox arises when a point null hypothesis is tested and an objective prior is sought for the alternative hypothesis. In particular, the posterior for the null hypothesis tends to one when the uncertainty, i.e. the variance, for the parameter value goes to infinity. We argue that the appropriate way to deal with the paradox is to use simple mathematics, and that any philosophical argument is to be regarded as irrelevant.
The Jeffreys-Lindley paradox displays how the use of a p-value (or number of standard deviations z) in a frequentist hypothesis test can lead to an inference that is radically different from that of a Bayesian hypothesis test in the form advocated by Harold Jeffreys in the 1930s and common today. The setting is the test of a well-specified null hypothesis (such as the Standard Model of elementary particle physics, possibly with "nuisance parameters") versus a composite alternative (such as the Standard Model plus a new force of nature of unknown strength). The p-value, as well as the ratio of …
There are three principle paradigms of statistics: Bayesian, frequentist and information-based inference. Although these paradigms are in agreement in some contexts, the Lindley paradox describes a class of problems, models of unknown dimension, where conflicting conclusions are generated by frequentist and Bayesian inference. This conflict can materially affect the scientific conclusions. Understanding the Lindley paradox---where it applies, why it occurs, and how it can be avoided---is therefore essential to the understanding of statistical analysis. In this paper, we revisit the Lindley par…
In the hypothesis testing framework, p-value is often computed to determine rejection of the null hypothesis or not. On the other hand, Bayesian approaches typically compute the posterior probability of the null hypothesis to evaluate its plausibility. We revisit Lindley's paradox (Lindley, 1957) and demystify the conflicting results between Bayesian and frequentist hypothesis testing procedures by casting a two-sided hypothesis as a combination of two one-sided hypotheses along the opposite directions. This can naturally circumvent the ambiguities of assigning a point mass to the null and cho…
In this article, we proposed an inverse Lindley distribution and studied its fundamental properties such as quantiles, mode, stochastic ordering and entropy measure. The proposed distribution is observed to be a heavy-tailed distribution and has a upside-down bathtub shape for its failure rate. Further, we proposed its applicability as a stress-strength reliability model for survival data analysis. The estimation of stress-strength parameters and $R=P[X>Y]$, the stress-strength reliability has been approached by both classical and Bayesian paradigms. Under Bayesian set-up, non-informative (Jef…
The Jeffreys-Lindley paradox exposes a rift between Bayesian and frequentist hypothesis testing that strikes at the heart of statistical inference. Contrary to what most current literature suggests, the paradox was central to the Bayesian testing methodology developed by Sir Harold Jeffreys in the late 1930s. Jeffreys showed that the evidence against a point-null hypothesis $\mathcal{H}_0$ scales with $\sqrt{n}$ and repeatedly argued that it would therefore be mistaken to set a threshold for rejecting $\mathcal{H}_0$ at a constant multiple of the standard error. Here we summarize Jeffreys's ea…
We propose a convergence mode for positive Radon measures which allows a sequence of probability measures to have an improper limiting measure. We define a sequence of vague priors as a sequence of probability measures that converges to an improper prior. We consider some cases where vague priors have necessarily large variances and other cases where they have not. We study the consequences of the convergence of prior distributions on the posterior analysis. Then we give some constructions of vague priors that approximate the Haar measures or the Jeffreys priors. We also revisit the Jeffreys-L…
Additive Bayesian networks are types of graphical models that extend the usual Bayesian generalized linear model to multiple dependent variables through the factorisation of the joint probability distribution of the underlying variables. When fitting an ABN model, the choice of the prior of the parameters is of crucial importance. If an inadequate prior - like a too weakly informative one - is used, data separation and data sparsity lead to issues in the model selection process. In this work a simulation study between two weakly and a strongly informative priors is presented. As weakly informa…
In 1933 Kolmogorov constructed a general theory that defines the modern concept of conditional expectation. In 1955 Renyi fomulated a new axiomatic theory for probability motivated by the need to include unbounded measures. We introduce a general concept of conditional expectation in Renyi spaces. In this theory improper priors are allowed, and the resulting posterior can also be improper. In 1965 Lindley published his classic text on Bayesian statistics using the theory of Renyi, but retracted this idea in 1973 due to the appearance of marginalization paradoxes presented by Dawid, Stone, and …
It has long been known that for the comparison of pairwise nested models, a decision based on the Bayes factor produces a consistent model selector (in the frequentist sense). Here we go beyond the usual consistency for nested pairwise models, and show that for a wide class of prior distributions, including intrinsic priors, the corresponding Bayesian procedure for variable selection in normal regression is consistent in the entire class of normal linear models. We find that the asymptotics of the Bayes factors for intrinsic priors are equivalent to those of the Schwarz (BIC) criterion. Also, …
We consider the specification of prior distributions for Bayesian model comparison, focusing on regression-type models. We propose a particular joint specification of the prior distribution across models so that sensitivity of posterior model probabilities to the dispersion of prior distributions for the parameters of individual models (Lindley's paradox) is diminished. We illustrate the behavior of inferential and predictive posterior quantities in linear and log-linear regressions under our proposed prior densities with a series of simulated and real data examples.
We consider a novel paradigm for Bayesian testing of hypotheses and Bayesian model comparison. Our alternative to the traditional construction of posterior probabilities that a given hypothesis is true or that the data originates from a specific model is to consider the models under comparison as components of a mixture model. We therefore replace the original testing problem with an estimation one that focus on the probability weight of a given model within a mixture model. We analyze the sensitivity on the resulting posterior distribution on the weights of various prior modeling on the weigh…
A key sticking point of Bayesian analysis is the choice of prior distribution, and there is a vast literature on potential defaults including uniform priors, Jeffreys' priors, reference priors, maximum entropy priors, and weakly informative priors. These methods, however, often manifest a key conceptual tension in prior modeling: a model encoding true prior information should be chosen without reference to the model of the measurement process, but almost all common prior modeling techniques are implicitly motivated by a reference likelihood. In this paper we resolve this apparent paradox by pl…
Prior distributions elicited for modelling the natural fluctuations or the uncertainty on parameters of Bayesian fishery population models, can be chosen among a vast range of statistical laws. Since the statistical framework is defined by observational processes, observational parameters enter into the estimation and must be considered random, similarly to parameters or states of interest like population levels or real catches. The former are thus perceived as nuisance parameters whose values are intrinsically linked to the considered experiment, which also require noninformative priors. In f…
This paper deals with a new Bayesian approach to the standard one-sample $z$- and $t$- tests. More specifically, let $x_1,\ldots,x_n$ be an independent random sample from a normal distribution with mean $μ$ and variance $σ^2$. The goal is to test the null hypothesis $\mathcal{H}_0: μ=μ_1$ against all possible alternatives. The approach is based on using the well-known formula of the Kullbak-Leibler divergence between two normal distributions (sampling and hypothesized distributions selected in an appropriate way). The change of the distance from a priori to a posteriori is compared through the…
In Bayesian hypothesis testing and model selection, prior distributions must be chosen carefully. For example, setting arbitrarily large prior scales for location parameters, which is common practice in estimation problems, can lead to undesirable behavior in testing (Lindley's paradox). We study the properties of some restricted type II maximum likelihood (type II ML) priors on regression coefficients. In type II ML, hyperparameters are "estimated" by maximizing the marginal likelihood of a model. In this article, we define priors by estimating their variances or covariance matrices, adding r…
The Posterior distribution of the Likelihood Ratio (PLR) is proposed by Dempster in 1974 for significance testing in the simple vs composite hypotheses case. In this hypotheses test case, classical frequentist and Bayesian hypotheses tests are irreconcilable, as emphasized by Lindley's paradox, Berger & Selke in 1987 and many others. However, Dempster shows that the PLR (with inner threshold 1) is equal to the frequentist p-value in the simple Gaussian case. In 1997, Aitkin extends this result by adding a nuisance parameter and showing its asymptotic validity under more general distributions. …
The development of prior distributions for Bayesian regression has traditionally been driven by the goal of achieving sensible model selection and parameter estimation. The formalization of properties that characterize good performance has led to the development and popularization of thick tailed mixtures of g priors such as the Zellner--Siow and hyper-g priors. The properties of a particular prior are typically illuminated under limits on the likelihood or the prior. In this paper we introduce a new, conditional information asymptotic that is motivated by the common data analysis setting wher…
Study of the bivariate normal distribution raises the full range of issues involving objective Bayesian inference, including the different types of objective priors (e.g., Jeffreys, invariant, reference, matching), the different modes of inference (e.g., Bayesian, frequentist, fiducial) and the criteria involved in deciding on optimal objective priors (e.g., ease of computation, frequentist performance, marginalization paradoxes). Summary recommendations as to optimal objective priors are made for a variety of inferences involving the bivariate normal distribution. In the course of the investi…
