We analyze the dynamics of an algorithm for approximate inference with large Gaussian latent variable models in a student-teacher scenario. To model nontrivial dependencies between the latent variables, we assume random covariance matrices drawn from rotation invariant ensembles. For the case of perfect data-model matching, the knowledge of static order parameters derived from the replica method allows us to obtain efficient algorithmic updates in terms of matrix-vector multiplications with a fixed matrix. Using the dynamical functional approach, we obtain an exact effective stochastic process in the thermodynamic limit for a single node. From this, we obtain closed-form expressions for the rate of the convergence. Analytical results are excellent agreement with simulations of single instances of large models.
Balliol College（贝利奥尔学院）；University of Oxford（牛津大学）
Bayesian inference has theoretical attractions as a principled framework for reasoning about beliefs. However, the motivations of Bayesian inference which claim it to be the only 'rational' kind of reasoning do not apply in practice. They create a binary split in which all approximate inference is equally 'irrational'. Instead, we should ask ourselves how to define a spectrum of more- and less-rational reasoning that explains why we might prefer one Bayesian approximation to another. I explore approximate inference in Bayesian neural networks and consider the unintended interactions between th…
We explain how effective automatic probability density function estimates can be constructed using contemporary Bayesian inference engines such as those based on no-U-turn sampling and expectation propagation. Extensive simulation studies demonstrate that the proposed density estimates have excellent comparative performance and scale well to very large sample sizes due to a binning strategy. Moreover, the approach is fully Bayesian and all estimates are accompanied by pointwise credible intervals. An accompanying package in the R language facilitates easy use of the new density estimates.
Cornell University（康奈尔大学）；Samsung AI Center in Moscow（三星莫斯科人工智能中心）；Higher School of Economics（高等经济学院）；Courant Institute and Center for Data Science, New York University（纽约大学库朗研究所与数据科学中心）
In this paper, we propose an analytical method for performing tractable approximate Gaussian inference (TAGI) in Bayesian neural networks. The method enables the analytical Gaussian inference of the posterior mean vector and diagonal covariance matrix for weights and biases. The method proposed has a computational complexity of $\mathcal{O}(n)$ with respect to the number of parameters $n$, and the tests performed on regression and classification benchmarks confirm that, for a same network architecture, it matches the performance of existing methods relying on gradient backpropagation.
University of Sydney（悉尼大学）；University of Science and Technology of China（中国科学技术大学）
The right to be forgotten has been legislated in many countries but the enforcement in machine learning would cause unbearable costs: companies may need to delete whole models learned from massive resources due to single individual requests. Existing works propose to remove the knowledge learned from the requested data via its influence function which is no longer naturally well-defined in Bayesian inference. This paper proposes a {\it Bayesian inference forgetting} (BIF) framework to realize the right to be forgotten in Bayesian inference. In the BIF framework, we develop forgetting algorithm…
Neural Architectures for Amortized Bayesian Inference: Statistical Foundations and Empirical Assessments
Department of Mathematics and Statistics, Indian Institute of Technology Kanpur（数学与统计学系，印度理工学院坎浦尔分校）；Marshall School of Business, University of Southern California（马歇尔商学院，南加州大学）
Cooperation is often implicitly assumed when learning from other agents. Cooperation implies that the agent selecting the data, and the agent learning from the data, have the same goal, that the learner infer the intended hypothesis. Recent models in human and machine learning have demonstrated the possibility of cooperation. We seek foundational theoretical results for cooperative inference by Bayesian agents through sequential data. We develop novel approaches analyzing consistency, rate of convergence and stability of Sequential Cooperative Bayesian Inference (SCBI). Our analysis of the eff…
Bayesian neural networks (BNN) can estimate the uncertainty in predictions, as opposed to non-Bayesian neural networks (NNs). However, BNNs have been far less widely used than non-Bayesian NNs in practice since they need iterative NN executions to predict a result for one data, and it gives rise to prohibitive computational cost. This computational burden is a critical problem when processing data streams with low-latency. To address this problem, we propose a novel model VQ-BNN, which approximates BNN inference for data streams. In order to reduce the computational burden, VQ-BNN inference pr…
Wellcome Centre for Human Neuroimaging, University College London（伦敦大学学院威康人类神经影像中心）；Department of Mathematics, Imperial College London（伦敦帝国理工学院数学系）
Active inference, a corollary of the free energy principle, is a formal way of describing the behavior of certain kinds of random dynamical systems that have the appearance of sentience. In this chapter, we describe how active inference combines Bayesian decision theory and optimal Bayesian design principles under a single imperative to minimize expected free energy. It is this aspect of active inference that allows for the natural emergence of information-seeking behavior. When removing prior outcomes preferences from expected free energy, active inference reduces to optimal Bayesian design, …
We present a new probabilistic method for detecting, deblending, and cataloging astronomical sources called the Bayesian Light Source Separator (BLISS). BLISS is based on deep generative models, which embed neural networks within a Bayesian model. For posterior inference, BLISS uses a new form of variational inference known as Forward Amortized Variational Inference. The BLISS inference routine is fast, requiring a single forward pass of the encoder networks on a GPU once the encoder networks are trained. BLISS can perform fully Bayesian inference on megapixel images in seconds, and produces h…
In Bayesian inference, an unknown measurement uncertainty is often quantified in terms of a Gamma distributed precision parameter, which is impractical when prior information on the standard deviation of the measurement uncertainty shall be utilised during inference. This paper thus introduces a method for transforming between a gamma distributed precision parameter and the distribution of the associated standard deviation. The proposed method is based on numerical optimisation and shows adequate results for a wide range of scenarios.
We introduce the concepts of Bayesian lens, characterizing the bidirectional structure of exact Bayesian inference, and statistical game, formalizing the optimization objectives of approximate inference problems. We prove that Bayesian inversions compose according to the compositional lens pattern, and exemplify statistical games with a number of classic statistical concepts, from maximum likelihood estimation to generalized variational Bayesian methods. This paper is the first in a series laying the foundations for a compositional account of the theory of active inference, and we therefore pa…
We discuss Bayesian inference for parameters selected using the data. First, we provide a critical analysis of the existing positions in the literature regarding the correct Bayesian approach under selection. Second, we propose two types of non-informative priors for selection models. These priors may be employed to produce a posterior distribution in the absence of prior information as well as to provide well-calibrated frequentist inference for the selected parameter. We test the proposed priors empirically in several scenarios.
We propose a quasi-Bayesian method to conduct inference for the production frontier. This approach combines multiple first-stage extreme quantile estimates by the quasi-Bayesian method to produce the point estimate and confidence interval for the production frontier. We show the asymptotic properties of the proposed estimator and the validity of the inference procedure. The finite sample performance of our method is illustrated through simulations and an empirical application.
Typical Bayesian inference requires parameter identification via likelihood parameterization, which has invited criticism for being less flexible than the Frequentist framework and subject to misspecification. Though misspecification may be avoided by functional parameter inference under a nonparametric model space, there does not exist a flexible Bayesian semiparametric model that would allow full control over the marginal prior over any general functional parameter. We present the technique of $θ$-augmentation which helps us manipulate nonparametric models into semiparametric ones that direc…
Multivariate functional data arise in a wide range of applications. One fundamental task is to understand the causal relationships among these functional objects of interest, which has not yet been fully explored. In this article, we develop a novel Bayesian network model for multivariate functional data where the conditional independence and causal structure are both encoded by a directed acyclic graph. Specifically, we allow the functional objects to deviate from Gaussian process, which is adopted by most existing functional data analysis models. The more reasonable non-Gaussian assumption i…
