In Bayesian hypothesis testing, evidence for a statistical model is quantified by the Bayes factor, which represents the relative likelihood of observed data under that model compared to another competing model. In general, computing Bayes factors is difficult, as computing the marginal likelihood of data under a given model requires integrating over a prior distribution of model parameters. In this paper, I capitalize on a particular choice of prior distribution that allows the Bayes factor to be expressed without integral representation and I develop a simple formula -- the Pearson Bayes factor -- that requires only minimal summary statistics commonly reported in scientific papers, such as the $t$ or $F$ score and the degrees of freedom. In addition to presenting this new result, I provide several examples of its use and report a simulation study validating its performance. Importantly, the Pearson Bayes factor gives applied researchers the ability to compute exact Bayes factors from minimal summary data, and thus easily assess the evidential value of any data for which these summary statistics are provided, even when the original data is not available.
We present a framework that enables estimation of low-dimensional sub-resolution reservoir properties directly from seismic data, without requiring the solution of a high dimensional seismic inverse problem. Our workflow is based on the Bayesian evidential learning approach and exploits learning the direct relation between seismic data and reservoir properties to efficiently estimate reservoir properties. The theoretical framework we develop allows incorporation of non-linear statistical models for seismic estimation problems. Uncertainty quantification is performed with Approximate Bayesian C…
Heidelberg University（海德堡大学）；Bosch Center for Artificial Intelligence（博世人工智能中心）
We propose a novel method for closed-form predictive distribution modeling with neural nets. In quantifying prediction uncertainty, we build on Evidential Deep Learning, which has been impactful as being both simple to implement and giving closed-form access to predictive uncertainty. We employ it to model aleatoric uncertainty and extend it to account also for epistemic uncertainty by converting it to a Bayesian Neural Net. While extending its uncertainty quantification capabilities, we maintain its analytically accessible predictive distribution model by performing progressive moment matchin…
We point out that the Neyman-Pearson lemma applies to Bayes factors if we consider expected type-1 and type-2 error rates. That is, the Bayes factor is the test statistic that maximises the expected power for a fixed expected type-1 error rate. For Bayes factors involving a simple null hypothesis, the expected type-1 error rate is just the completely frequentist type-1 error rate. Lastly we remark on connections between the Karlin-Rubin theorem and uniformly most powerful tests, and Bayes factors. This provides frequentist motivations for computing the Bayes factor and could help reconcile Bay…
German Aerospace Center (DLR)（德国航空航天中心）；Institute for Simulation Intelligence（模拟智能研究所）
There is significant need for principled uncertainty reasoning in machine learning systems as they are increasingly deployed in safety-critical domains. A new approach with uncertainty-aware neural networks (NNs), based on learning evidential distributions for aleatoric and epistemic uncertainties, shows promise over traditional deterministic methods and typical Bayesian NNs, yet several important gaps in the theory and implementation of these networks remain. We discuss three issues with a proposed solution to extract aleatoric and epistemic uncertainties from regression-based neural networks…
Pearson's correlation is one of the most common measures of linear dependence. Recently, Bernardo (2015) introduced a flexible class of priors to study this measure in a Bayesian setting. For this large class of priors we show that the (marginal) posterior for Pearson's correlation coefficient and all of the posterior moments are analytic. Our results are available in the open-source software package JASP.
Comparing competing mathematical models of complex natural processes is a shared goal among many branches of science. The Bayesian probabilistic framework offers a principled way to perform model comparison and extract useful metrics for guiding decisions. However, many interesting models are intractable with standard Bayesian methods, as they lack a closed-form likelihood function or the likelihood is computationally too expensive to evaluate. With this work, we propose a novel method for performing Bayesian model comparison using specialized deep learning architectures. Our method is purely …
We analyze hypotheses tests using classical results on large deviations to compare two models, each one described by a different Hölder Gibbs probability measure. One main difference to the classical hypothesis tests in Decision Theory is that here the two measures are singular with respect to each other. Among other objectives, we are interested in the decay rate of the wrong decisions probability, when the sample size $n$ goes to infinity. We show a dynamical version of the Neyman-Pearson Lemma displaying the ideal test within a certain class of similar tests. This test becomes exponentially…
Deterministic neural networks (NNs) are increasingly being deployed in safety critical domains, where calibrated, robust, and efficient measures of uncertainty are crucial. In this paper, we propose a novel method for training non-Bayesian NNs to estimate a continuous target as well as its associated evidence in order to learn both aleatoric and epistemic uncertainty. We accomplish this by placing evidential priors over the original Gaussian likelihood function and training the NN to infer the hyperparameters of the evidential distribution. We additionally impose priors during training such th…
Department of Chemical Engineering, University of Michigan（美国密歇根大学化学工程系）；Michigan Institute for Data & AI in Society, University of Michigan（美国密歇根大学数据与社会人工智能研究所）；Department of Statistics, University of Michigan（美国密歇根大学统计系）
Toward Efficient Uncertainty in LLMs through Evidential Knowledge Distillation
Department of Computer Science and Engineering, Indian Institute of Technology Hyderabad（计算机科学与工程系，印度理工学院海得拉巴分校）；Jagiellonian University, Faculty of Mathematics and Computer Science（雅盖隆大学，数学与计算机科学学院）
Bayesian inference is limited in scope because it cannot be applied in idealized contexts where none of the hypotheses under consideration is true and because it is committed to always using the likelihood as a measure of evidential favoring, even when that is inappropriate. The purpose of this paper is to study inductive inference in a very general setting where finding the truth is not necessarily the goal and where the measure of evidential favoring is not necessarily the likelihood. I use an accuracy argument to argue for probabilism and I develop a new kind of argument to argue for two ge…
Deargen Inc.（Deargen 公司）；Deargen USA Inc.（Deargen 美国公司）
The Evidential regression network (ENet) estimates a continuous target and its predictive uncertainty without costly Bayesian model averaging. However, it is possible that the target is inaccurately predicted due to the gradient shrinkage problem of the original loss function of the ENet, the negative log marginal likelihood (NLL) loss. In this paper, the objective is to improve the prediction accuracy of the ENet while maintaining its efficient uncertainty estimation by resolving the gradient shrinkage problem. A multi-task learning (MTL) framework, referred to as MT-ENet, is proposed to acco…
Ozyegin University（奥兹耶金大学）；US Army Research Lab（美国陆军研究实验室）；Bosch Center for Artificial Intelligence（博世人工智能中心）
Deterministic neural nets have been shown to learn effective predictors on a wide range of machine learning problems. However, as the standard approach is to train the network to minimize a prediction loss, the resultant model remains ignorant to its prediction confidence. Orthogonally to Bayesian neural nets that indirectly infer prediction uncertainty through weight uncertainties, we propose explicit modeling of the same using the theory of subjective logic. By placing a Dirichlet distribution on the class probabilities, we treat predictions of a neural net as subjective opinions and learn t…
In this paper, I develop a formula for estimating Bayes factors directly from minimal summary statistics produced in repeated measures analysis of variance designs. The formula, which requires knowing only the $F$-statistic, the number of subjects, and the number of repeated measurements per subject, is based on the BIC approximation of the Bayes factor, a common default method for Bayesian computation with linear models. In addition to providing computational examples, I report a simulation study in which I demonstrate that the formula compares favorably to a recently developed, more complex …
Empirical Bayes methods have been around for a long time and have a wide range of applications. These methods provide a way in which historical data can be aggregated to provide estimates of the posterior mean. This thesis revisits some of the empirical Bayesian methods and develops new applications. We first look at a linear empirical Bayes estimator and apply it on ranking and symbolic data. Next, we consider Tweedie's formula and show how it can be applied to analyze a microarray dataset. The application of the formula is simplified with the Pearson system of distributions. Saddlepoint appr…
This article describes an extension of classical χ^2 goodness-of-fit tests to Bayesian model assessment. The extension, which essentially involves evaluating Pearson's goodness-of-fit statistic at a parameter value drawn from its posterior distribution, has the important property that it is asymptotically distributed as a χ^2 random variable on K-1 degrees of freedom, independently of the dimension of the underlying parameter vector. By examining the posterior distribution of this statistic, global goodness-of-fit diagnostics are obtained. Advantages of these diagnostics include ease of interp…
We provide a geometric interpretation to Bayesian inference that allows us to introduce a natural measure of the level of agreement between priors, likelihoods, and posteriors. The starting point for the construction of our geometry is the simple observation that the marginal likelihood can be regarded as an inner product between the prior and the likelihood. A key concept in our geometry is that of compatibility, a measure which is based on the same construction principles as Pearson correlation, but which can be used to assess how much the prior agrees with the likelihood, to gauge the sensi…
A Bayesian Reinterpretation of Cornfield-Type Sensitivity Analysis: From Thresholds to Probabilities
GCS-fMRI, Koelliker Hospital and Department of Psychology, University of Turin（都灵大学心理系 GCS-fMRI 科埃利凯尔医院）；FOCUS Laboratory, Department of Psychology, University of Turin（都灵大学心理系 FOCUS 实验室）；Neuroscience Institute of Turin (NIT)（都灵神经科学研究所）
Alzheimer's disease (AD) is characterized by a sequence of pathological changes, which are commonly assessed in vivo using MRI and PET. Currently, the most approaches to analyze statistical associations between brain regions rely on Pearson correlation. However, these are prone to spurious correlations arising from uninformative shared variance. Notably, there are no appropriate multivariate statistical models available that can easily integrate dozens of variables derived from such data, being able to use the additional information provided from the combination of data sources. Gaussian graph…
L.J. Savage once hoped to show that "the superficially incompatible systems of ideas associated on the one hand with [subjective Bayesianism] and on the other hand with [classical statistics]...lend each other mutual support and clarification." By 1972, however, he had largely "lost faith in the devices" of classical statistics. One aspect of those "devices" that he found objectionable is that differences among the "stopping rules" that are used to decide when to end an experiment which are "noninformative" from a Bayesian perspective can affect decisions made using a classical approach. Two e…
Feature selection aims to select the smallest subset of features for a specified level of performance. The optimal achievable classification performance on a feature subset is summarized by its Receiver Operating Curve (ROC). When infinite data is available, the Neyman- Pearson (NP) design procedure provides the most efficient way of obtaining this curve. In practice the design procedure is applied to density estimates from finite data sets. We perform a detailed statistical analysis of the resulting error propagation on finite alphabets. We show that the estimated performance curve (EPC) prod…
Siemens AG（西门子股份公司）；Technical University of Munich（慕尼黑工业大学）
To facilitate a wide-spread acceptance of AI systems guiding decision making in real-world applications, trustworthiness of deployed models is key. That is, it is crucial for predictive models to be uncertainty-aware and yield well-calibrated (and thus trustworthy) predictions for both in-domain samples as well as under domain shift. Recent efforts to account for predictive uncertainty include post-processing steps for trained neural networks, Bayesian neural networks as well as alternative non-Bayesian approaches such as ensemble approaches and evidential deep learning. Here, we propose an ef…
The Posterior distribution of the Likelihood Ratio (PLR) is proposed by Dempster in 1974 for significance testing in the simple vs composite hypotheses case. In this hypotheses test case, classical frequentist and Bayesian hypotheses tests are irreconcilable, as emphasized by Lindley's paradox, Berger & Selke in 1987 and many others. However, Dempster shows that the PLR (with inner threshold 1) is equal to the frequentist p-value in the simple Gaussian case. In 1997, Aitkin extends this result by adding a nuisance parameter and showing its asymptotic validity under more general distributions. …
We compute exact values respectively bounds of "distances" - in the sense of (transforms of) power divergences and relative entropy - between two discrete-time Galton-Watson branching processes with immigration GWI for which the offspring as well as the immigration is arbitrarily Poisson-distributed (leading to arbitrary type of criticality). Implications for asymptotic distinguishability behaviour in terms of contiguity and entire separation of the involved GWI are given, too. Furthermore, we determine the corresponding limit quantities for the context in which the two GWI converge to Feller-…
We propose new summary measures of diagnostic test accuracy which can be used as companions to existing diagnostic accuracy measures. Conceptually, our summary measures are tantamount to the so-called Hellinger affinity and we show that they can be regarded as measures of agreement constructed from similar geometrical principles as Pearson correlation. A covariate-specific version of our summary index is developed, which can be used to assess the discrimination performance of a diagnostic test, conditionally on the value of a predictor. Nonparametric Bayes estimators for the proposed indexes a…
Calibrated and uncertain? Evaluating uncertainty estimates in binary classification models
Department of Computer science, Electrical engineering and Mathematical sciences Western Norway University of Applied Sciences（计算机科学、电气工程和数学科学系西部挪威应用科学大学）；Department of Informatics University of Bergen（信息学院卑尔根大学）
MIT Lincoln Laboratory（麻省理工学院林肯实验室）；Harvard University（哈佛大学）
Network detection is an important capability in many areas of applied research in which data can be represented as a graph of entities and relationships. Oftentimes the object of interest is a relatively small subgraph in an enormous, potentially uninteresting background. This aspect characterizes network detection as a "big data" problem. Graph partitioning and network discovery have been major research areas over the last ten years, driven by interest in internet search, cyber security, social networks, and criminal or terrorist activities. The specific problem of network discovery is addres…
