We showed how to use trained neural networks to perform Bayesian reasoning in order to solve tasks outside their initial scope. Deep generative models provide prior knowledge, and classification/regression networks impose constraints. The tasks at hand were formulated as Bayesian inference problems, which we approximately solved through variational or sampling techniques. The approach built on top of already trained networks, and the addressable questions grew super-exponentially with the number of available networks. In its simplest form, the approach yielded conditional generative models. However, multiple simultaneous constraints constitute elaborate questions. We compared the approach to specifically trained generators, showed how to solve riddles, and demonstrated its compatibility with state-of-the-art architectures.
Bayesian network classifiers are used in many fields, and one common class of classifiers are naive Bayes classifiers. In this paper, we introduce an approach for reasoning about Bayesian network classifiers in which we explicitly convert them into Ordered Decision Diagrams (ODDs), which are then used to reason about the properties of these classifiers. Specifically, we present an algorithm for converting any naive Bayes classifier into an ODD, and we show theoretically and experimentally that this algorithm can give us an ODD that is tractable in size even given an intractable number of insta…
We present the Bayesian Case Model (BCM), a general framework for Bayesian case-based reasoning (CBR) and prototype classification and clustering. BCM brings the intuitive power of CBR to a Bayesian generative framework. The BCM learns prototypes, the "quintessential" observations that best represent clusters in a dataset, by performing joint inference on cluster labels, prototypes and important features. Simultaneously, BCM pursues sparsity by learning subspaces, the sets of features that play important roles in the characterization of the prototypes. The prototype and subspace representation…
Bayesian networks offer great potential for use in automating large scale diagnostic reasoning tasks. Gibbs sampling is the main technique used to perform diagnostic reasoning in large richly interconnected Bayesian networks. Unfortunately Gibbs sampling can take an excessive time to generate a representative sample. In this paper we describe and test a number of heuristic strategies for improving sampling in noisy-or Bayesian networks. The strategies include Monte Carlo Markov chain sampling techniques other than Gibbs sampling. Emphasis is put on strategies that can be implemented in distrib…
National University of Singapore（新加坡国立大学）；Massachusetts Institute of Technology（麻省理工学院）
This paper presents a recursive reasoning formalism of Bayesian optimization (BO) to model the reasoning process in the interactions between boundedly rational, self-interested agents with unknown, complex, and costly-to-evaluate payoff functions in repeated games, which we call Recursive Reasoning-Based BO (R2-B2). Our R2-B2 algorithm is general in that it does not constrain the relationship among the payoff functions of different agents and can thus be applied to various types of games such as constant-sum, general-sum, and common-payoff games. We prove that by reasoning at level 2 or more a…
Radboud Universiteit（拉德堡德大学）；University College London（伦敦大学学院）
This chapter offers an accessible introduction to the channel-based approach to Bayesian probability theory. This framework rests on algebraic and logical foundations, inspired by the methodologies of programming language semantics. It offers a uniform, structured and expressive language for describing Bayesian phenomena in terms of familiar programming concepts, like channel, predicate transformation and state transformation. The introduction also covers inference in Bayesian networks, which will be modelled by a suitable calculus of string diagrams.
University of Duisburg-Essen（杜伊斯堡-埃森大学）；University of Leicester（莱斯特大学）
This paper exploits extended Bayesian networks for uncertainty reasoning on Petri nets, where firing of transitions is probabilistic. In particular, Bayesian networks are used as symbolic representations of probability distributions, modelling the observer's knowledge about the tokens in the net. The observer can study the net by monitoring successful and failed steps. An update mechanism for Bayesian nets is enabled by relaxing some of their restrictions, leading to modular Bayesian nets that can conveniently be represented and modified. As for every symbolic representation, the question is h…
Carnegie Mellon University（卡内基梅隆大学）；Santa Fe Institute（圣塔菲研究所）
Recent work in cognitive science has uncovered a diversity of explanatory values, or dimensions along which we judge explanations as better or worse. We propose a Bayesian account of how these values fit together to guide explanation. The resulting taxonomy provides a set of predictors for which explanations people prefer and shows how core values from psychology, statistics, and the philosophy of science emerge from a common mathematical framework. In addition to operationalizing the explanatory virtues associated with, for example, scientific argument-making, this framework also enables us t…
Doctors often rely on their past experience in order to diagnose patients. For a doctor with enough experience, almost every patient would have similarities to key cases seen in the past, and each new patient could be viewed as a mixture of these key past cases. Because doctors often tend to reason this way, an efficient computationally aided diagnostic tool that thinks in the same way might be helpful in locating key past cases of interest that could assist with diagnosis. This article develops a novel mathematical model to mimic the type of logical thinking that physicians use when consideri…
Balliol College（贝利奥尔学院）；University of Oxford（牛津大学）
Bayesian inference has theoretical attractions as a principled framework for reasoning about beliefs. However, the motivations of Bayesian inference which claim it to be the only 'rational' kind of reasoning do not apply in practice. They create a binary split in which all approximate inference is equally 'irrational'. Instead, we should ask ourselves how to define a spectrum of more- and less-rational reasoning that explains why we might prefer one Bayesian approximation to another. I explore approximate inference in Bayesian neural networks and consider the unintended interactions between th…
Université de Toulouse（图卢兹大学）；INRAE（法国国家农业食品与环境研究院）；AgroParisTech（巴黎高科农业学院）；Univ. Paris-Saclay（巴黎萨克雷大学）
Bayesian networks are probabilistic graphical models with a wide range of application areas including gene regulatory networks inference, risk analysis and image processing. Learning the structure of a Bayesian network (BNSL) from discrete data is known to be an NP-hard task with a superexponential search space of directed acyclic graphs. In this work, we propose a new polynomial time algorithm for discovering a subset of all possible cluster cuts, a greedy algorithm for approximately solving the resulting linear program, and a generalised arc consistency algorithm for the acyclicity constrain…
The notion of Craig interpolant, used as a form of explanation in automated reasoning, is adapted from logical inference to statistical inference and used to explain inferences made by neural networks. The method produces explanations that are at the same time concise, understandable and precise.
University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）；University of Bristol（布里斯托大学）
We develop ShiftMatch, a new training-data-dependent likelihood for robustness to corruption in Bayesian neural networks (BNNs). ShiftMatch is inspired by the training-data-dependent "EmpCov" priors from Izmailov et al. (2021a), and efficiently matches test-time spatial correlations to those at training time. Critically, ShiftMatch is designed to leave the neural network's training time likelihood unchanged, allowing it to use publicly available samples from pre-trained BNNs. Using pre-trained HMC samples, ShiftMatch gives strong performance improvements on CIFAR-10-C, outperforms EmpCov prior…
Bayesian Reasoning Enabled by Spin-Orbit Torque Magnetic Tunnel Junctions
Institute of Physics, University of Chinese Academy of Sciences, Chinese Academy of Sciences（中国科学院大学 中国科学院物理研究所）；Center of Materials Science and Optoelectronics Engineering, University of Chinese Academy of Sciences（中国科学院大学 材料科学与光电技术学院）；Songshan Lake Materials Laboratory（松山湖材料实验室）；Physical Science and Engineering Division, King Abdullah University of Science and Technology (KAUST)（阿卜杜拉国王科技大学 物理科学与工程学院）
Columbia University（哥伦比亚大学）；Columbia University School of Professional Studies（哥伦比亚大学专业研究学院）；Department of Statistics（统计学系）；Columbia University Department of Computer Science（哥伦比亚大学计算机科学系）
Bayesian Reasoning for Physics Informed Neural Networks
Institute for Theoretical Physics, University of Wroc aw（沃拉夫大学理论物理研究所）；Institute of Low Temperature and Structure Research（低温与结构研究所）；Polish Academy of Sciences（波兰科学院）
Many resources for forensic scholars and practitioners, such as journal articles, guidance documents, and textbooks, address how to make a value of evidence assessment in the form of a likelihood ratio (LR) when deciding between two competing propositions. These texts often describe experts presenting their LR values to other parties in the judicial system, such as lawyers, judges, and potentially jurors, but few texts explicitly address how a recipient is expected to utilize the provided LR value. Those that do often imply, or directly suggest, a hybrid modification of Bayes' rule in which a …
This paper modifies Jaynes's axioms of plausible reasoning and derives the minimum relative entropy principle, Bayes's rule, as well as maximum likelihood from first principles. The new axioms, which I call the Optimum Information Principle, is applicable whenever the decision maker is given the data and the relevant background information. These axioms provide an answer to the question "why maximize entropy when faced with incomplete information?"
Monash University（莫纳什大学）；Bayesian Intelligence（贝叶斯智能公司）；Automatic Studio（奥托马蒂克工作室）；University of Strathclyde（思克莱德大学）；University College London（伦敦大学学院）
In many complex, real-world situations, problem solving and decision making require effective reasoning about causation and uncertainty. However, human reasoning in these cases is prone to confusion and error. Bayesian networks (BNs) are an artificial intelligence technology that models uncertain situations, supporting probabilistic and causal reasoning and decision making. However, to date, BN methodologies and software require significant upfront training, do not provide much guidance on the model building process, and do not support collaboratively building BNs. BARD (Bayesian ARgumentation…
A theory of quantitative inference about the parameters of sampling distributions is constructed deductively by following very general rules, referred to as the Cox-Polya-Jaynes Desiderata. The inferences are made in terms of probability distributions that are assigned to the parameters. The Desiderata, focusing primarily on consistency of the plausible reasoning, lead to unique assignments of these probabilities in the case of sampling distributions that are invariant under Lie groups. In the scalar cases, e.g. in the case of inferring a single location or scale parameter, the requirement for…
COBRA-PPM: A Causal Bayesian Reasoning Architecture Using Probabilistic Programming for Robot Manipulation Under Uncertainty
Oxford Robotics Institute, Dept. Engineering Science, University of Oxford, UK（牛津大学机器人研究所）；Microsoft Research, Redmond, WA, USA（微软研究院）；Bristol Robotics Laboratory, School of Engineering, University of the West of England, Bristol, UK（布里斯托尔机器人实验室）
We propose a Bayesian evidence framework to facilitate transfer learning from pre-trained deep convolutional neural networks (CNNs). Our framework is formulated on top of a least squares SVM (LS-SVM) classifier, which is simple and fast in both training and testing, and achieves competitive performance in practice. The regularization parameters in LS-SVM is estimated automatically without grid search and cross-validation by maximizing evidence, which is a useful measure to select the best performing CNN out of multiple candidates for transfer learning; the evidence is optimized efficiently by …
Georg-August University of Göttingen（哥廷根乔治-奥古斯特大学）；University of Tübingen（图宾根大学）
A recent paper by Liu et al. combines the topics of adversarial training and Bayesian Neural Networks (BNN) and suggests that adversarially trained BNNs are more robust against adversarial attacks than their non-Bayesian counterparts. Here, I analyze the proposed defense and suggest that one needs to adjust the adversarial attack to incorporate the stochastic nature of a Bayesian network to perform an accurate evaluation of its robustness. Using this new type of attack I show that there appears to be no strong evidence for higher robustness of the adversarially trained BNNs.
McGill University（麦吉尔大学）；Universita di Pavia（帕维亚大学）
Bayesian Belief Networks (BBNs) are a powerful formalism for reasoning under uncertainty but bear some severe limitations: they require a large amount of information before any reasoning process can start, they have limited contradiction handling capabilities, and their ability to provide explanations for their conclusion is still controversial. There exists a class of reasoning systems, called Truth Maintenance Systems (TMSs), which are able to deal with partially specified knowledge, to provide well-founded explanation for their conclusions, and to detect and handle contradictions. TMSs inco…
Stochastic sampling algorithms, while an attractive alternative to exact algorithms in very large Bayesian network models, have been observed to perform poorly in evidential reasoning with extremely unlikely evidence. To address this problem, we propose an adaptive importance sampling algorithm, AIS-BN, that shows promising convergence rates even under extreme conditions and seems to outperform the existing sampling algorithms consistently. Three sources of this performance improvement are (1) two heuristics for initialization of the importance function that are based on the theoretical proper…
University of Oxford（牛津大学）；Indiana University（印第安纳大学）
We draw a formal connection between using synthetic training data to optimize neural network parameters and approximate, Bayesian, model-based reasoning. In particular, training a neural network using synthetic data can be viewed as learning a proposal distribution generator for approximate inference in the synthetic-data generative model. We demonstrate this connection in a recognition task where we develop a novel Captcha-breaking architecture and train it using synthetic data, demonstrating both state-of-the-art performance and a way of computing task-specific posterior uncertainty. Using a…
Google Brain（谷歌大脑）；Microsoft Research AI（微软人工智能研究院）；University of Cambridge（剑桥大学）
There is a previously identified equivalence between wide fully connected neural networks (FCNs) and Gaussian processes (GPs). This equivalence enables, for instance, test set predictions that would have resulted from a fully Bayesian, infinitely wide trained FCN to be computed without ever instantiating the FCN, but by instead evaluating the corresponding GP. In this work, we derive an analogous equivalence for multi-layer convolutional neural networks (CNNs) both with and without pooling layers, and achieve state of the art results on CIFAR10 for GPs without trainable kernels. We also introd…
Make Me a BNN: A Simple Strategy for Estimating Bayesian Uncertainty from Pre-trained Models
U2IS ENSTA Paris Institut Polytechnique de Paris（巴黎综合理工学院ENSTA巴黎高等先进技术学院U2IS实验室）；Université Paris-Saclay（巴黎萨克雷大学）；NVIDIA（英伟达）；National University of Singapore（新加坡国立大学）
Queen Mary University of London（伦敦玛丽女王大学）；School of Electronic Engineering and Computer Science（电子工程与计算机科学学院）
Bayesian Networks (BNs) are graphical probabilistic models that have proven popular in medical applications. While numerous medical BNs have been published, most are presented fait accompli without explanation of how the network structure was developed or justification of why it represents the correct structure for the given medical application. This means that the process of building medical BNs from experts is typically ad hoc and offers little opportunity for methodological improvement. This paper proposes generally applicable and reusable medical reasoning patterns to aid those developing …
Credit risk modelling is an integral part of the global financial system. While there has been great attention paid to neural network models for credit default prediction, such models often lack the required interpretation mechanisms and measures of the uncertainty around their predictions. This work develops and compares Bayesian Neural Networks(BNNs) for credit card default modelling. This includes a BNNs trained by Gaussian approximation and the first implementation of BNNs trained by Hybrid Monte Carlo(HMC) in credit risk modelling. The results on the Taiwan Credit Dataset show that BNNs w…
Many researchers want to unify probability and logic by defining logical probability or probabilistic logic reasonably. This paper tries to unify statistics and logic so that we can use both statistical probability and logical probability at the same time. For this purpose, this paper proposes the P-T probability framework, which is assembled with Shannon's statistical probability framework for communication, Kolmogorov's probability axioms for logical probability, and Zadeh's membership functions used as truth functions. Two kinds of probabilities are connected by an extended Bayes' theorem, …
The MITRE Corporation（MITRE公司）；George Mason University（乔治梅森大学）
This paper examines the use of Bayesian Networks to tackle one of the tougher problems in requirements engineering, translating user requirements into system requirements. The approach taken is to model domain knowledge as Bayesian Network fragments that are glued together to form a complete view of the domain specific system requirements. User requirements are introduced as evidence and the propagation of belief is used to determine what are the appropriate system requirements as indicated by user requirements. This concept has been demonstrated in the development of a system specification an…
A major reason behind the success of probability calculus is that it possesses a number of valuable tools, which are based on the notion of probabilistic independence. In this paper, I identify a notion of logical independence that makes some of these tools available to a class of propositional databases, called argument databases. Specifically, I suggest a graphical representation of argument databases, called argument networks, which resemble Bayesian networks. I also suggest an algorithm for reasoning with argument networks, which resembles a basic algorithm for reasoning with Bayesian netw…
Institute of Information Theory and Automation of the AS CR（捷克科学院信息理论与自动化研究所）
A difficult task in modeling with Bayesian networks is the elicitation of numerical parameters of Bayesian networks. A large number of parameters is needed to specify a conditional probability table (CPT) that has a larger parent set. In this paper we show that, most CPTs from real applications of Bayesian networks can actually be very well approximated by tables that require substantially less parameters. This observation has practical consequence not only for model elicitation but also for efficient probabilistic reasoning with these networks.
Recent developments show that Multiply Sectioned Bayesian Networks (MSBNs) can be used for diagnosis of natural systems as well as for model-based diagnosis of artificial systems. They can be applied to single-agent oriented reasoning systems as well as multi-agent distributed probabilistic reasoning systems. Belief propagation between a pair of subnets plays a central role in maintenance of global consistency in a MSBN. This paper studies the operation UpdateBelief, presented originally with MSBNs, for inter-subnet propagation. We analyze how the operation achieves its intended functionality,…
Socrates-Mol: Self-Oriented Cognitive Reasoning through Autonomous Trial-and-Error with Empirical-Bayesian Screening for Molecules
State Key Laboratory of Heavy Oil Processing, China University of Petroleum (Beijing), Beijing 102249, China（石油加工国家重点实验室，中国石油大学（北京））；Shanghai AI Laboratory,L1 Building, International Media Port, No. 129 Longwen Road, Xuhui District, Shanghai（上海人工智能实验室）；College of Energy Innovation, China University of Petroleum (Beijing), Beijing 102249, China（能源创新学院，中国石油大学（北京））
This paper presents a simple framework for Horn clause abduction, with probabilities associated with hypotheses. It is shown how this representation can represent any probabilistic knowledge representable in a Bayesian belief network. The main contributions are in finding a relationship between logical and probabilistic notions of evidential reasoning. This can be used as a basis for a new way to implement Bayesian Networks that allows for approximations to the value of the posterior probabilities, and also points to a way that Bayesian networks can be extended beyond a propositional language.
Open University of the Netherlands（荷兰开放大学）；Osaka University（大阪大学）；Radboud University（拉德堡德大学）
Explaining predictions from Bayesian networks, for example to physicians, is non-trivial. Various explanation methods for Bayesian network inference have appeared in literature, focusing on different aspects of the underlying reasoning. While there has been a lot of technical research, there is very little known about how well humans actually understand these explanations. In this paper, we present ongoing research in which four different explanation approaches were compared through a survey by asking a group of human participants to interpret the explanations.
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；School of Information, Renmin University of China（中国人民大学信息学院）；Beijing Academy of Artificial Intelligence（北京智源人工智能研究院）
Commonsense reasoning in natural language is a desired ability of artificial intelligent systems. For solving complex commonsense reasoning tasks, a typical solution is to enhance pre-trained language models~(PTMs) with a knowledge-aware graph neural network~(GNN) encoder that models a commonsense knowledge graph~(CSKG). Despite the effectiveness, these approaches are built on heavy architectures, and can't clearly explain how external knowledge resources improve the reasoning capacity of PTMs. Considering this issue, we conduct a deep empirical analysis, and find that it is indeed relation fe…
In this paper, we consider Hybrid Mixed Networks (HMN) which are Hybrid Bayesian Networks that allow discrete deterministic information to be modeled explicitly in the form of constraints. We present two approximate inference algorithms for HMNs that integrate and adjust well known algorithmic principles such as Generalized Belief Propagation, Rao-Blackwellised Importance Sampling and Constraint Propagation to address the complexity of modeling and reasoning in HMNs. We demonstrate the performance of our approximate inference algorithms on randomly generated HMNs.
Idioms are small, reusable Bayesian network (BN) fragments that represent generic types of uncertain reasoning. This paper shows how idioms can be used to build causal BNs for product safety and risk assessment that use a combination of data and knowledge. We show that the specific product safety idioms that we introduce are sufficient to build full BN models to evaluate safety and risk for a wide range of products. The resulting models can be used by safety regulators and product manufacturers even when there are limited (or no) product testing data.
University of Michigan AI Laboratory（密歇根大学人工智能实验室）
Qualitative probabilistic reasoning in a Bayesian network often reveals tradeoffs: relationships that are ambiguous due to competing qualitative influences. We present two techniques that combine qualitative and numeric probabilistic reasoning to resolve such tradeoffs, inferring the qualitative relationship between nodes in a Bayesian network. The first approach incrementally marginalizes nodes that contribute to the ambiguous qualitative relationships. The second approach evaluates approximate Bayesian networks for bounds of probability distributions, and uses these bounds to determinate qua…
Northrop Research and Technology Center（诺斯罗普研究与技术中心）；University of Southern California（南加州大学）
This paper discuses multiple Bayesian networks representation paradigms for encoding asymmetric independence assertions. We offer three contributions: (1) an inference mechanism that makes explicit use of asymmetric independence to speed up computations, (2) a simplified definition of similarity networks and extensions of their theory, and (3) a generalized representation scheme that encodes more types of asymmetric independence assertions than do similarity networks.
We provide four case studies that use Bayesian machinery to making inductive reasoning. Our main motivation relies in offering several instances where the Bayesian approach to data analysis is exploited at its best to perform complex tasks, such as description, testing, estimation, and prediction. This work is not meant to be either a reference text or a survey in Bayesian statistical inference. Our goal is simply to provide several examples that use Bayesian methodology to solve data-driven problems. The topics we cover here, include problems in Bayesian nonparametrics, Bayesian analysis of t…
It is a challenge for any Knowledge Base reasoning to manage ubiquitous uncertain ontology as well as uncertain updating times, while achieving acceptable service levels at minimum computational cost. This paper proposes an application-independent merging ontologies for any open interaction system. A solution that uses Multi-Entity Bayesan Networks with SWRL rules, and a Java program is presented to dynamically monitor Exogenous and Endogenous temporal evolution on updating merging ontologies on a probabilistic framework for the Semantic Web.
School of Electronic Engineering & Computer Science, Queen Mary University of London（伦敦玛丽女王大学电子工程与计算机科学学院）；Centre for Trauma Sciences, Blizard Institute, Queen Mary University of London（伦敦玛丽女王大学布利扎德研究所创伤科学中心）；Royal London Hospital, Barts and The London NHS Trust（皇家伦敦医院）；Academic Department of Military Surgery and Trauma, Royal Centre for Defence Medicine（皇家国防医学中心军事外科与创伤学术部）
Various AI models are increasingly being considered as part of clinical decision-support tools. However, the trustworthiness of such models is rarely considered. Clinicians are more likely to use a model if they can understand and trust its predictions. Key to this is if its underlying reasoning can be explained. A Bayesian network (BN) model has the advantage that it is not a black-box and its reasoning can be explained. In this paper, we propose an incremental explanation of inference that can be applied to hybrid BNs, i.e. those that contain both discrete and continuous nodes. The key quest…
