While deep neural networks provide good performance for a range of challenging tasks, calibration and uncertainty estimation remain major challenges, especially under distribution shift. In this paper, we propose the amortized conditional normalized maximum likelihood (ACNML) method as a scalable general-purpose approach for uncertainty estimation, calibration, and out-of-distribution robustness with deep networks. Our algorithm builds on the conditional normalized maximum likelihood (CNML) coding scheme, which has minimax optimal properties according to the minimum description length principle, but is computationally intractable to evaluate exactly for all but the simplest of model classes. We propose to use approximate Bayesian inference technqiues to produce a tractable approximation to the CNML distribution. Our approach can be combined with any approximate inference algorithm that provides tractable posterior densities over model parameters. We demonstrate that ACNML compares favorably to a number of prior techniques for uncertainty estimation in terms of calibration on out-of-distribution inputs.
Radboud University（拉德堡德大学）；University of Utrecht（乌得勒支大学）；University of Amsterdam（阿姆斯特丹大学）
In this paper, we introduce a new form of amortized variational inference by using the forward KL divergence in a joint-contrastive variational loss. The resulting forward amortized variational inference is a likelihood-free method as its gradient can be sampled without bias and without requiring any evaluation of either the model joint distribution or its derivatives. We prove that our new variational loss is optimized by the exact posterior marginals in the fully factorized mean-field approximation, a property that is not shared with the more conventional reverse KL inference. Furthermore, w…
Laboratoire de Physique de l’École normale supérieure（巴黎高等师范学院物理实验室）；CNRS（法国国家科学研究中心）；PSL University（巴黎文理研究大学）；Sorbonne Université（索邦大学）；Université Paris Cité（巴黎西岱大学）；Max Planck Institute for Dynamics and Self-organization（马克斯·普朗克动力学与自组织研究所）；University of Washington（华盛顿大学）；Fred Hutchinson Cancer Research Center（弗雷德·哈钦森癌症研究中心）
Simulation-based inference enables learning the parameters of a model even when its likelihood cannot be computed in practice. One class of methods uses data simulated with different parameters to infer models of the likelihood-to-evidence ratio, or equivalently the posterior function. Here we frame the inference task as an estimation of an energy function parametrized with an artificial neural network. We present an intuitive approach where the optimal model of the likelihood-to-evidence ratio is found by maximizing the likelihood of simulated data. Within this framework, the connection betwe…
Amortized Simulation-Based Inference in Generalized Bayes via Neural Posterior Estimation
Department of Statistics, University of Oxford, Oxford, United Kingdom（英国牛津大学统计系）；Department of Statistics, University of Auckland, Auckland, New Zealand（新西兰奥克兰大学统计系）
Generalized Bayesian Inference for Scientific Simulators via Amortized Cost Estimation
Tübingen AI Center（图宾根人工智能中心）；Machine Learning in Science, Excellence Cluster Machine Learning, University of Tübingen（图宾根大学机器学习卓越集群科学机器学习系）；Department Empirical Inference, Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所经验推断系）
Overcoming Selection Bias in Statistical Studies With Amortized Bayesian Inference
University of Bonn（波恩大学）；Institut Pasteur（巴斯德研究所）；University of Freiburg（弗赖堡大学）；LMU University Hospital（慕尼黑大学附属医院）；German Center for Infection Research（德国感染研究中心）；Fraunhofer Institute ITMP（弗劳恩霍夫ITMP研究所）；Max von Pettenkofer Institute（马克斯·冯·佩滕科弗研究所）；LMU Munich（慕尼黑大学）；German Research Center for Environmental Health（德国环境健康研究中心）；Institut Necker-Enfants Malades（内克-儿童疾病研究所）
Scalable Vertical Federated Learning via Data Augmentation and Amortized Inference
Centre for Data Science, Queensland University of Technology（昆士兰科技大学数据科学中心）；School of Mathematical Sciences, Queensland University of Technology（昆士兰科技大学数学科学学院）；Euler Institute, Università della Svizzera italiana (USI)（瑞士意大利语区大学欧拉研究所）；Insubria University（因苏布里亚大学）
Comparing competing mathematical models of complex natural processes is a shared goal among many branches of science. The Bayesian probabilistic framework offers a principled way to perform model comparison and extract useful metrics for guiding decisions. However, many interesting models are intractable with standard Bayesian methods, as they lack a closed-form likelihood function or the likelihood is computationally too expensive to evaluate. With this work, we propose a novel method for performing Bayesian model comparison using specialized deep learning architectures. Our method is purely …
Aalto University（阿尔托大学）；Inserm（法国国家健康与医学研究院）；Bordeaux Population Health Vaccine Research Institute（波尔多人口健康疫苗研究所）；Université de Bordeaux（波尔多大学）；Inria Bordeaux Sud-ouest（法国国家信息与自动化研究所波尔多西南分部）
The Predictive Normalized Maximum Likelihood (pNML) scheme has been recently suggested for universal learning in the individual setting, where both the training and test samples are individual data. The goal of universal learning is to compete with a ``genie'' or reference learner that knows the data values, but is restricted to use a learner from a given model class. The pNML minimizes the associated regret for any possible value of the unknown label. Furthermore, its min-max regret can serve as a pointwise measure of learnability for the specific training and data sample. In this work we exa…
Yale University（耶鲁大学）；University Helsinki（赫尔辛基大学）；Nara Institute of Science and Technology（奈良先端科学技术大学院大学）
The normalized maximized likelihood (NML) provides the minimax regret solution in universal data compression, gambling, and prediction, and it plays an essential role in the minimum description length (MDL) method of statistical modeling and estimation. Here we show that the normalized maximum likelihood has a Bayes-like representation as a mixture of the component models, even in finite samples, though the weights of linear combination may be both positive and negative. This representation addresses in part the relationship between MDL and Bayes modeling. This representation has the advantage…
Neural Architectures for Amortized Bayesian Inference: Statistical Foundations and Empirical Assessments
Department of Mathematics and Statistics, Indian Institute of Technology Kanpur（数学与统计学系，印度理工学院坎浦尔分校）；Marshall School of Business, University of Southern California（马歇尔商学院，南加州大学）
From Mice to Trains: Amortized Bayesian Inference on Graph Data
Department of Statistics, TU Dortmund University（统计系，多特蒙德技术大学）；School of Public Health, Imperial College London（公共卫生学院，伦敦帝国理工学院）；Department of Biology, University of Oxford（生物学系，牛津大学）；Department of Mechanical Engineering, Karlsruhe Institute of Technology（机械工程系，卡尔斯鲁厄理工学院）
Cluster of Excellence SimTech（SimTech卓越集群）；University of Stuttgart（斯图加特大学）；Heidelberg University（海德堡大学）；Cluster of Excellence STRUCTURES（STRUCTURES卓越集群）
Neural density estimators have proven remarkably powerful in performing efficient simulation-based Bayesian inference in various research domains. In particular, the BayesFlow framework uses a two-step approach to enable amortized parameter estimation in settings where the likelihood function is implicitly defined by a simulation program. But how faithful is such inference when simulations are poor representations of reality? In this paper, we conceptualize the types of model misspecification arising in simulation-based inference and systematically investigate the performance of the BayesFlow …
University of Liège（列日大学）；University of Vienna（维也纳大学）
Posterior inference with an intractable likelihood is becoming an increasingly common task in scientific domains which rely on sophisticated computer simulations. Typically, these forward models do not admit tractable densities forcing practitioners to make use of approximations. This work introduces a novel approach to address the intractability of the likelihood and the marginal model. We achieve this by learning a flexible amortized estimator which approximates the likelihood-to-evidence ratio. We demonstrate that the learned ratio estimator can be embedded in MCMC samplers to approximate l…
University of Tübingen（蒂宾根大学）；Tübingen AI Center（蒂宾根人工智能中心）；University of British Columbia（不列颠哥伦比亚大学）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
We compared Bayes factors to normalized maximum likelihood for the simple case of selecting between an order-constrained versus a full binomial model. This comparison revealed two qualitative differences in testing order constraints regarding data dependence and model preference.
In this work we consider data-driven optimization problems where one must maximize a function given only queries at a fixed set of points. This problem setting emerges in many domains where function evaluation is a complex and expensive process, such as in the design of materials, vehicles, or neural network architectures. Because the available data typically only covers a small manifold of the possible space of inputs, a principal challenge is to be able to construct algorithms that can reason about uncertainty and out-of-distribution values, since a naive optimizer can easily exploit an esti…
Although with progress in introducing auxiliary amortized inference models, learning discrete latent variable models is still challenging. In this paper, we show that the annoying difficulty of obtaining reliable stochastic gradients for the inference model and the drawback of indirectly optimizing the target log-likelihood can be gracefully addressed in a new method based on stochastic approximation (SA) theory of the Robbins-Monro type. Specifically, we propose to directly maximize the target log-likelihood and simultaneously minimize the inclusive divergence between the posterior and the in…
Microsoft Research, Redmond（微软研究院，雷德蒙德）；Duke University（杜克大学）；Brown University（布朗大学）
Collaborative filtering is widely used in modern recommender systems. Recent research shows that variational autoencoders (VAEs) yield state-of-the-art performance by integrating flexible representations from deep neural networks into latent variable models, mitigating limitations of traditional linear factor models. VAEs are typically trained by maximizing the likelihood (MLE) of users interacting with ground-truth items. While simple and often effective, MLE-based training does not directly maximize the recommendation-quality metrics one typically cares about, such as top-N ranking. In this …
We reveal the relations between the conditional normalized maximum likelihood (CNML) distributions and Bayesian predictive densities based on the latent information priors (LIPs). In particular, CNML3, which is one type of CNML distributions, is investigated. The Bayes projection of a predictive density, which is an information projection of the predictive density on a set of Bayesian predictive densities, is considered. We prove that the sum of the Bayes projection divergence of CNML3 and the conditional mutual information is asymptotically constant. This result implies that the Bayes project…
Fast and Robust Likelihood-Guided Diffusion Posterior Sampling with Amortized Variational Inference
Institute of Foundation Models, MBZUAI（基础模型研究所，MBZUAI）；Huawei Lagrange Mathematics and Computing Research Center, Paris, France（华为拉格朗日数学与计算研究中心，法国巴黎）；Mohamed Bin Zayed University of AI, Chair of the ML Department, EPITA, Laboratoire Recherche de l'EPITA（穆罕默德·本·扎耶德人工智能大学，机器学习系主任，EPITA，EPITA研究实验室）
Detecting out-of-distribution (OOD) samples is vital for developing machine learning based models for critical safety systems. Common approaches for OOD detection assume access to some OOD samples during training which may not be available in a real-life scenario. Instead, we utilize the {\em predictive normalized maximum likelihood} (pNML) learner, in which no assumptions are made on the tested input. We derive an explicit expression of the pNML and its generalization error, denoted as the {\em regret}, for a single layer neural network (NN). We show that this learner generalizes well when (i…
Korea Advanced Institute of Science and Technology（韩国科学技术院）；University of Seoul（首尔市立大学）；ioCrops（ioCrops公司）；EWHA Womans University（梨花女子大学）
Bayesian inference without the likelihood evaluation, or likelihood-free inference, has been a key research topic in simulation studies for gaining quantitatively validated simulation models on real-world datasets. As the likelihood evaluation is inaccessible, previous papers train the amortized neural network to estimate the ground-truth posterior for the simulation of interest. Training the network and accumulating the dataset alternatively in a sequential manner could save the total simulation budget by orders of magnitude. In the data accumulation phase, the new simulation inputs are chose…
We propose a number of new algorithms for learning deep energy models and demonstrate their properties. We show that our SteinCD performs well in term of test likelihood, while SteinGAN performs well in terms of generating realistic looking images. Our results suggest promising directions for learning better models by combining GAN-style methods with traditional energy-based learning.
Several approximate inference methods have been proposed for deep discrete latent variable models. However, non-parametric methods which have previously been successfully employed for classical sparse coding models have largely been unexplored in the context of deep models. We propose a non-parametric iterative algorithm for learning discrete latent representations in such deep models. Additionally, to learn scale invariant discrete features, we propose local data scaling variables. Lastly, to encourage sparsity in our representations, we propose a Beta-Bernoulli process prior on the latent fa…
The elicitation of power priors, based on the availability of historical data, is realized by raising the likelihood function of the historical data to a fractional power δ, which quantifies the degree of discounting of the historical information in making inference with the current data. When δ is not pre-specified and is treated as random, it can be estimated from the data using Bayesian updating paradigm. However, in the original form of the joint power prior Bayesian approach, certain positive constants before the likelihood of the historical data could be multiplied when different setting…
Due to the intractable partition function, training energy-based models (EBMs) by maximum likelihood requires Markov chain Monte Carlo (MCMC) sampling to approximate the gradient of the Kullback-Leibler divergence between data and model distributions. However, it is non-trivial to sample from an EBM because of the difficulty of mixing between modes. In this paper, we propose to learn a variational auto-encoder (VAE) to initialize the finite-step MCMC, such as Langevin dynamics that is derived from the energy function, for efficient amortized sampling of the EBM. With these amortized MCMC sampl…
In many areas of science, complex phenomena are modeled by stochastic parametric simulators, often featuring high-dimensional parameter spaces and intractable likelihoods. In this context, performing Bayesian inference can be challenging. In this work, we present a novel method that enables amortized inference over arbitrary subsets of the parameters, without resorting to numerical integration, which makes interpretation of the posterior more convenient. Our method is efficient and can be implemented with arbitrary neural network architectures. We demonstrate the applicability of the method on…
A model of Poissonian observation having a jump (change-point) in the intensity function is considered. Two cases are studied. The first one corresponds to the situation when the jump size converges to a non-zero limit, while in the second one the limit is zero. The limiting likelihood ratios in these two cases are quite different. In the first case, like in the case of a fixed jump size, the normalized likelihood ratio converges to a log Poisson process. In the second case, the normalized likelihood ratio converges to a log Wiener process, and so, the statistical problems of parameter estimat…
Samsung AI Center（三星人工智能中心）；Rutgers University（罗格斯大学）
Inference networks of traditional Variational Autoencoders (VAEs) are typically amortized, resulting in relatively inaccurate posterior approximation compared to instance-wise variational optimization. Recent semi-amortized approaches were proposed to address this drawback; however, their iterative gradient update procedures can be computationally demanding. To address these issues, in this paper we introduce an accurate amortized inference algorithm. We propose a novel recursive mixture estimation algorithm for VAEs that iteratively augments the current mixture with new components so as to ma…
