Estimating the entropy rate of discrete time series is a challenging problem with important applications in numerous areas including neuroscience, genomics, image processing and natural language processing. A number of approaches have been developed for this task, typically based either on universal data compression algorithms, or on statistical estimators of the underlying process distribution. In this work, we propose a fully-Bayesian approach for entropy estimation. Building on the recently introduced Bayesian Context Trees (BCT) framework for modelling discrete time series as variable-memory Markov chains, we show that it is possible to sample directly from the induced posterior on the entropy rate. This can be used to estimate the entire posterior distribution, providing much richer information than point estimates. We develop theoretical results for the posterior distribution of the entropy rate, including proofs of consistency and asymptotic normality. The practical utility of the method is illustrated on both simulated and real-world data, where it is found to outperform state-of-the-art alternatives.
In many scientific fields, researchers are interested in discovering features with substantial effect on the response from a large number of features while controlling the proportion of false discoveries. By incorporating the knockoff procedure in the Bayesian framework, we develop the Bayesian knockoff filter (BKF) for selecting features that have important effect on the response. In contrast to the fixed knockoff variables in a frequentist procedure, we allow the knockoff variables to be continuously updated using the Markov chain Monte Carlo. Based on the posterior samples and the elaborate…
Lund University（隆德大学）；University of Freiburg（弗赖堡大学）；Bosch Center for Artificial Intelligence（博世人工智能中心）；Stanford University（斯坦福大学）；DBtune（DBtune公司）
Information-theoretic Bayesian optimization techniques have become popular for optimizing expensive-to-evaluate black-box functions due to their non-myopic qualities. Entropy Search and Predictive Entropy Search both consider the entropy over the optimum in the input space, while the recent Max-value Entropy Search considers the entropy over the optimal value in the output space. We propose Joint Entropy Search (JES), a novel information-theoretic acquisition function that considers an entirely new quantity, namely the entropy over the joint optimal probability density over both input and outp…
Osaka University（大阪大学）；Japan Fine Ceramics Center（日本精细陶瓷中心）；Nagoya Institute of Technology（名古屋工业大学）；National Institute for Material Science（国立材料科学研究所）
This paper studies an entropy-based multi-objective Bayesian optimization (MBO). The entropy search is successful approach to Bayesian optimization. However, for MBO, existing entropy-based methods ignore trade-off among objectives or introduce unreliable approximations. We propose a novel entropy-based MBO called Pareto-frontier entropy search (PFES) by considering the entropy of Pareto-frontier, which is an essential notion of the optimality of the multi-objective problem. Our entropy can incorporate the trade-off relation of the optimal values, and further, we derive an analytical formula w…
Dirichlet Bayesian Network Scores and the Maximum Relative Entropy Principle
从数据中学习贝叶斯网络的经典方法是识别最大后验（MAP）网络结构。在离散贝叶斯网络的情况下，MAP网络是通过最大化几个可能的贝叶斯狄利克雷（BD）分数之一来选择的;最著名的是Heckerman et al（1995）的贝叶斯狄利克雷等价均匀（BDeu）分数。BDeu的关键属性来自其对网络中每个局部分布参数的一致先验，这使得结构学习计算效率高;它不需要从专家那里获取先验知识;并且它满足分数等价性。在本文中，我们将审查BD分数的推导和性质，特别是BDeu，我们将把它们与相应的熵估计联系起来，从信息论的角度研究它们。为此，我们将在Giffin和Caticha（2007）的基础工作的背景下工作，他们表明贝叶斯推理可以被框定为最大相对熵原理的特殊情况。我们将使用这种联系来证明BDeu不应该用于稀疏数据的结构学习，因为它违反了最大相对熵原则;从更经典的贝叶斯模型选择的角度来看，它也是有问题的，因为它产生的贝叶斯因子对其唯一的超参数值敏感。使用大型模拟研究，我们在以前的工作中发现（Scutari，2016）贝叶斯Dirichlet稀疏（BD）得分似乎在结构学习中提供了更好的准确性;在本文中，我们进一步表明BD不会受到上述问题的影响，我们建议将其用于稀疏数据而不是BDeu。最后，将表明，这些问题实际上是同一问题的不同方面和先验分布假设的结果。
Local Entropy Search over Descent Sequences for Bayesian Optimization
Institute for Data Science in Mechanical Engineering, RWTH Aachen University, Germany（机械工程数据科学研究所，亚琛工业大学，德国）；Learning Systems and Robotics Lab, Technical University of Munich, Germany（学习系统与机器人实验室，慕尼黑技术大学，德国）
Bosch Center for Artificial Intelligence（博世人工智能中心）；ETH Zürich（苏黎世联邦理工学院）
We consider the problem of robust optimization within the well-established Bayesian optimization (BO) framework. While BO is intrinsically robust to noisy evaluations of the objective function, standard approaches do not consider the case of uncertainty about the input parameters. In this paper, we propose Noisy-Input Entropy Search (NES), a novel information-theoretic acquisition function that is designed to find robust optima for problems with both input and measurement noise. NES is based on the key insight that the robust objective in many cases can be modeled as a Gaussian process, howeve…
Bayesian optimization (BO) is a model-based approach to sequentially optimize expensive black-box functions, such as the validation error of a deep neural network with respect to its hyperparameters. In many real-world scenarios, the optimization is further subject to a priori unknown constraints. For example, training a deep network configuration may fail with an out-of-memory error when the model is too large. In this work, we focus on a general formulation of Gaussian process-based BO with continuous or binary constraints. We propose constrained Max-value Entropy Search (cMES), a novel info…
We present MESMOC+, an improved version of Max-value Entropy search for Multi-Objective Bayesian optimization with Constraints (MESMOC). MESMOC+ can be used to solve constrained multi-objective problems when the objectives and the constraints are expensive to evaluate. MESMOC+ works by minimizing the entropy of the solution of the optimization problem in function space, i.e., the Pareto frontier, to guide the search for the optimum. The cost of MESMOC+ is linear in the number of objectives and constraints. Furthermore, it is often significantly smaller than the cost of alternative methods base…
We propose a modification of the improved cross entropy (iCE) method to enhance its performance for network reliability assessment. The iCE method performs a transition from the nominal density to the optimal importance sampling (IS) density via a parametric distribution model whose cross entropy with the optimal IS is minimized. The efficiency and accuracy of the iCE method are largely influenced by the choice of the parametric model. In the context of reliability of systems with independent multi-state components, the obvious choice of the parametric family is the categorical distribution. W…
Many real-world problems can be phrased as a multi-objective optimization problem, where the goal is to identify the best set of compromises between the competing objectives. Multi-objective Bayesian optimization (BO) is a sample efficient strategy that can be deployed to solve these vector-valued optimization problems where access is limited to a number of noisy objective function evaluations. In this paper, we propose a novel information-theoretic acquisition function for BO called Joint Entropy Search (JES), which considers the joint information gain for the optimal set of inputs and output…
Bayesian Entropy Neural Networks for Physics-Aware Prediction
School for Engineering of Matter, Transport and Energy（物质、运输与能源工程学院）；Arizona State University（亚利桑那州立大学）；School of Computing and Augmented Intelligence（计算与增强智能学院）
Gradient-Based Approximate Bayesian Inference with Entropy-Optimized Summary Statistics for Compartmental Models
School of Mathematics and Statistics, University of St Andrews（圣安德鲁斯大学数学与统计学院）；Centre for Research into Ecological and Environmental Modelling, University of St Andrews（圣安德鲁斯大学生态与环境建模研究中心）
National University of Singapore（新加坡国立大学）；MIT（麻省理工学院）
Although the existing max-value entropy search (MES) is based on the widely celebrated notion of mutual information, its empirical performance can suffer due to two misconceptions whose implications on the exploration-exploitation trade-off are investigated in this paper. These issues are essential in the development of future acquisition functions and the improvement of the existing ones as they encourage an accurate measure of the mutual information such as the rectified MES (RMES) acquisition function we develop in this work. Unlike the evaluation of MES, we derive a closed-form probability…
In inverse problems, the parameters of a model are estimated based on observations of the model response. The Bayesian approach is powerful for solving such problems; one formulates a prior distribution for the parameter state that is updated with the observations to compute the posterior parameter distribution. Solving for the posterior distribution can be challenging when, e.g., prior and posterior significantly differ from one another and/or the parameter space is high-dimensional. We use a sequence of importance sampling measures that arise by tempering the likelihood to approach inverse p…
Robust Entropy Search for Safe Efficient Bayesian Optimization
Fraunhofer Institute for Intelligent Analysis and Information Systems IAIS（弗劳恩霍夫智能分析与信息系统研究所）；Federal Institute for Materials Research and Testing BAM（联邦材料研究与测试研究所）；University of Applied Sciences Bonn-Rhein-Sieg（波恩-莱茵-锡格应用科学大学）；Fraunhofer Institute for Machine Tools and Forming Technology IWU（弗劳恩霍夫机床与成形技术研究所）；TUD Dresden University of Technology（德累斯顿工业大学）；Institute of Mechatronic Engineering（机电工程研究所）
Gifu University（岐阜大学）；Nagoya Institute of Technology（名古屋工业大学）；Nagoya University（名古屋大学）；Japan Science and Technology Agency（日本科学技术振兴机构）；National Institute for Material Science（国立材料科学研究所）；RIKEN Center for Advanced Intelligence Project（理化学研究所先进智能项目中心）
In a standard setting of Bayesian optimization (BO), the objective function evaluation is assumed to be highly expensive. Multi-fidelity Bayesian optimization (MFBO) accelerates BO by incorporating lower fidelity observations available with a lower sampling cost. In this paper, we focus on the information-based approach, which is a popular and empirically successful approach in BO. For MFBO, however, existing information-based methods are plagued by difficulty in estimating the information gain. We propose an approach based on max-value entropy search (MES), which greatly facilitates computati…
We consider the problem of constrained multi-objective blackbox optimization using expensive function evaluations, where the goal is to approximate the true Pareto set of solutions satisfying a set of constraints while minimizing the number of function evaluations. For example, in aviation power system design applications, we need to find the designs that trade-off total energy and the mass while satisfying specific thresholds for motor temperature and voltage of cells. This optimization requires performing expensive computational simulations to evaluate designs. In this paper, we propose a ne…
Scarce data is a major challenge to scaling robot learning to truly complex tasks, as we need to generalize locally learned policies over different "contexts". Bayesian optimization approaches to contextual policy search (CPS) offer data-efficient policy learning that generalize over a context space. We propose to improve data-efficiency by factoring typically considered contexts into two components: target-type contexts that correspond to a desired outcome of the learned behavior, e.g. target position for throwing a ball; and environment type contexts that correspond to some state of the envi…
A Bayesian General Linear Modeling Approach to Cortical Surface fMRI Data Analysis
皮质表面功能磁共振成像（cs-fMRI）最近越来越受欢迎，与传统的体积功能磁共振成像，因为它允许更有意义的空间平滑，更符合贝叶斯空间模型的各向同性和平稳性的共同假设。然而，由于尚未针对cs-fMRI数据提出贝叶斯空间模型，因此大多数分析继续采用经典的逐体素通用线性模型（GLM）（Worsley and Friston 1995）。在这里，我们提出了一个贝叶斯GLM CS-fMRI，它采用了一类复杂的空间过程，灵活地模拟潜在的激活字段。我们使用集成嵌套拉普拉斯近似（INLA），这是一种高度准确和高效的贝叶斯计算技术（Rue et al. 2009）。为了识别激活区域，我们提出了一种基于潜在场联合后验分布的偏移集方法，该方法消除了多重比较校正的需要。最后，我们提出了一种新的贝叶斯方法进行多学科分析，以解决现有文献中的差距。通过仿真研究和人类连接组计划的运动任务fMRI研究，验证了该方法并与经典GLM进行了比较。所提出的贝叶斯方法导致更平滑的激活估计，更准确的假阳性控制，并提高了检测真正活跃区域的能力。
National University of Singapore（新加坡国立大学）；Massachusetts Institute of Technology（麻省理工学院）
Information-based Bayesian optimization (BO) algorithms have achieved state-of-the-art performance in optimizing a black-box objective function. However, they usually require several approximations or simplifying assumptions (without clearly understanding their effects on the BO performance) and/or their generalization to batch BO is computationally unwieldy, especially with an increasing batch size. To alleviate these issues, this paper presents a novel trusted-maximizers entropy search (TES) acquisition function: It measures how much an input query contributes to the information gain on the …
Real-world problems often involve the optimization of several objectives under multiple constraints. An example is the hyper-parameter tuning problem of machine learning algorithms. In particular, the minimization of the estimation of the generalization error of a deep neural network and at the same time the minimization of its prediction time. We may also consider as a constraint that the deep neural network must be implemented in a chip with an area below some size. Here, both the objectives and the constraint are black boxes, i.e., functions whose analytical expressions are unknown and are …
This article reviews how organisms learn and recognize the world through the dynamics of neural networks from the perspective of Bayesian inference, and introduces a view on how such dynamics is described by the laws for the entropy of neural activity, a paradigm that we call thermodynamics of the Bayesian brain. The Bayesian brain hypothesis sees the stimulus-evoked activity of neurons as an act of constructing the Bayesian posterior distribution based on the generative model of the external world that an organism possesses. A closer look at the stimulus-evoked activity at early sensory corti…
Gradient Dynamics of Attention: How Cross-Entropy Sculpts Bayesian Manifolds
Columbia University（哥伦比亚大学）；Columbia University School of Professional Studies（哥伦比亚大学专业研究学院）；Department of Statistics（统计学系）；Columbia University Department of Computer Science（哥伦比亚大学计算机科学系）
Bosch Center for AI（博世人工智能中心）；Idiap Research Institute（Idiap研究院）；National University of Singapore（新加坡国立大学）；University of Cambridge（剑桥大学）
Scarce data is a major challenge to scaling robot learning to truly complex tasks, as we need to generalize locally learned policies over different task contexts. Contextual policy search offers data-efficient learning and generalization by explicitly conditioning the policy on a parametric context space. In this paper, we further structure the contextual policy representation. We propose to factor contexts into two components: target contexts that describe the task objectives, e.g. target position for throwing a ball; and environment contexts that characterize the environment, e.g. initial po…
