With the increasing demand for interpretability in machine learning, functional ANOVA decomposition has gained renewed attention as a principled tool for breaking down high-dimensional function into low-dimensional components that reveal the contributions of different variable groups. Recently, Tensor Product Neural Network (TPNN) has been developed and applied as basis functions in the functional ANOVA model, referred to as ANOVA-TPNN. A disadvantage of ANOVA-TPNN, however, is that the components to be estimated must be specified in advance, which makes it difficult to incorporate higher-order TPNNs into the functional ANOVA model due to computational and memory constraints. In this work, we propose Bayesian-TPNN, a Bayesian inference procedure for the functional ANOVA model with TPNN basis functions, enabling the detection of higher-order components with reduced computational cost compared to ANOVA-TPNN. We develop an efficient MCMC algorithm and demonstrate that Bayesian-TPNN performs well by analyzing multiple benchmark datasets. Theoretically, we prove that the posterior of Bayesian-TPNN is consistent.
The Gaussian process is a standard tool for building emulators for both deterministic and stochastic computer experiments. However, application of Gaussian process models is greatly limited in practice, particularly for large-scale and many-input computer experiments that have become typical. We propose a multi-resolution functional ANOVA model as a computationally feasible emulation alternative. More generally, this model can be used for large-scale and many-input non-linear regression problems. An overlapping group lasso approach is used for estimation, ensuring computational feasibility in …
The transition from non-renewable to renewable energies represents a global societal challenge, and developing a sustainable energy portfolio is an especially daunting task for developing countries where little to no information is available regarding the abundance of renewable resources such as wind. Weather model simulations are key to obtain such information when observational data are scarce and sparse over a country as large and geographically diverse as Saudi Arabia. However, output from such models is uncertain, as it depends on inputs such as the parametrization of the physical process…
A Model Intercomparison Project (MIP) consists of teams who each estimate the same underlying quantity (e.g., temperature projections to the year 2070), and the spread of the estimates indicates their uncertainty. It recognizes that a community of scientists will not agree completely but that there is value in looking for a consensus and information in the range of disagreement. A simple average of the teams' outputs gives a consensus estimate, but it does not recognize that some outputs are more variable than others. Statistical analysis of variance (ANOVA) models offer a way to obtain a weig…
Gaussian Process (GP) models are a class of flexible non-parametric models that have rich representational power. By using a Gaussian process with additive structure, complex responses can be modelled whilst retaining interpretability. Previous work showed that additive Gaussian process models require high-dimensional interaction terms. We propose the orthogonal additive kernel (OAK), which imposes an orthogonality constraint on the additive functions, enabling an identifiable, low-dimensional representation of the functional relationship. We connect the OAK kernel to functional ANOVA decompos…
Bayesian Fisher's Discriminant for Functional Data
Institute of Information Science, Academia Sinica（中央研究院资讯科学研究所）；Institute of Statistics, National Chung-Hsing University（中兴大学统计学研究所）；Institute of Networking and Multimedia, National Taiwan University（台湾大学网路与多媒体研究所）
University of Toronto（多伦多大学）；Vector Institute（向量研究所）；Tsinghua University（清华大学）
Variational Bayesian neural networks (BNNs) perform variational inference over weights, but it is difficult to specify meaningful priors and approximate posteriors in a high-dimensional weight space. We introduce functional variational Bayesian neural networks (fBNNs), which maximize an Evidence Lower BOund (ELBO) defined directly on stochastic processes, i.e. distributions over functions. We prove that the KL divergence between stochastic processes equals the supremum of marginal KL divergences over all finite sets of inputs. Based on this, we introduce a practical training objective which ap…
Bayesian neural networks provide a direct and natural way to extend standard deep neural networks to support probabilistic deep learning through the use of probabilistic layers that, traditionally, encode weight (and bias) uncertainty. In particular, hybrid Bayesian neural networks utilize standard deterministic layers together with few probabilistic layers judicially positioned in the networks for uncertainty estimation. A major aspect and benefit of Bayesian inference is that priors, in principle, provide the means to encode prior knowledge for use in inference and prediction. However, it is…
Deep Functional Factor Models: Forecasting High-Dimensional Functional Time Series via Bayesian Nonparametric Factorization
London School of Economics（伦敦经济学院）；Faculty of Business（商学院）；Economics, The University of Hong Kong（经济学，香港大学）；Management School, University of Liverpool（利物浦大学管理学院）
The Bayesian treatment of neural networks dictates that a prior distribution is specified over their weight and bias parameters. This poses a challenge because modern neural networks are characterized by a large number of parameters, and the choice of these priors has an uncontrolled effect on the induced functional prior, which is the distribution of the functions obtained by sampling the parameters from their prior distribution. We argue that this is a hugely limiting aspect of Bayesian deep learning, and this work tackles this limitation in a practical and effective way. Our proposal is to …
Deakin University（迪肯大学）；Centre for Pattern Recognition and Data Analytics（模式识别与数据分析中心）；Institute for Frontier Materials（前沿材料研究所）
Real world experiments are expensive, and thus it is important to reach a target in minimum number of experiments. Experimental processes often involve control variables that changes over time. Such problems can be formulated as a functional optimisation problem. We develop a novel Bayesian optimisation framework for such functional optimisation of expensive black-box processes. We represent the control function using Bernstein polynomial basis and optimise in the coefficient space. We derive the theory and practice required to dynamically adjust the order of the polynomial degree, and show ho…
Applied Artificial Intelligence Institute (A2I2)（应用人工智能研究所（A2I2））；Deakin University（迪肯大学）
We propose an algorithm for Bayesian functional optimisation - that is, finding the function to optimise a process - guided by experimenter beliefs and intuitions regarding the expected characteristics (length-scale, smoothness, cyclicity etc.) of the optimal solution encoded into the covariance function of a Gaussian Process. Our algorithm generates a sequence of finite-dimensional random subspaces of functional space spanned by a set of draws from the experimenter's Gaussian Process. Standard Bayesian optimisation is applied on each subspace, and the best solution found used as a starting po…
Tensor Product Neural Networks for Functional ANOVA Model
Department of Statistics, Seoul National University, Seoul, Republic of Korea（统计系，首尔国立大学）；Department of Applied Mathematics, University of Twente, Enschede, Netherlands（应用数学系，埃因霍温理工大学）；Samsung Electronics Co., Ltd, Republic of Korea（三星电子有限公司）
West Virginia University（西弗吉尼亚大学）；Rochester Institute of Technology（罗切斯特理工学院）
A promising approach for scalable Gaussian processes (GPs) is the Karhunen-Loève (KL) decomposition, in which the GP kernel is represented by a set of basis functions which are the eigenfunctions of the kernel operator. Such decomposed kernels have the potential to be very fast, and do not depend on the selection of a reduced set of inducing points. However KL decompositions lead to high dimensionality, and variable selection becomes paramount. This paper reports a new method of forward variable selection, enabled by the ordered nature of the basis functions in the KL expansion of the Bayesian…
University of Oxford（牛津大学）；Alan Turing Institute（艾伦·图灵研究所）；University of Birmingham（伯明翰大学）；University of Manchester（曼彻斯特大学）
Variational Autoencoders (VAEs) have become a popular approach for dimensionality reduction. However, despite their ability to identify latent low-dimensional structures embedded within high-dimensional data, these latent representations are typically hard to interpret on their own. Due to the black-box nature of VAEs, their utility for healthcare and genomics applications has been limited. In this paper, we focus on characterising the sources of variation in Conditional VAEs. Our goal is to provide a feature-level variance decomposition, i.e. to decompose variation in the data by separating o…
Sparse Bayesian Deep Functional Learning with Structured Region Selection
School of Statistics and Data Science, Shanghai University of Finance and Economics（上海金融学院统计与数据科学学院）；School of Statistics（统计学院）；Data Science, Shanghai University of Finance（金融大学数据科学学院）；Department of Biostatistics, Yale School of Public Health（耶鲁大学公共卫生学院生物统计学系）
We analyze the dynamics of an algorithm for approximate inference with large Gaussian latent variable models in a student-teacher scenario. To model nontrivial dependencies between the latent variables, we assume random covariance matrices drawn from rotation invariant ensembles. For the case of perfect data-model matching, the knowledge of static order parameters derived from the replica method allows us to obtain efficient algorithmic updates in terms of matrix-vector multiplications with a fixed matrix. Using the dynamical functional approach, we obtain an exact effective stochastic process…
Bayesian neural networks (BNNs) can account for both aleatoric and epistemic uncertainty. However, in BNNs the priors are often specified over the weights which rarely reflects true prior knowledge in large and complex neural network architectures. We present a simple approach to incorporate prior knowledge in BNNs based on external summary information about the predicted classification probabilities for a given dataset. The available summary information is incorporated as augmented data and modeled with a Dirichlet process, and we derive the corresponding \emph{Summary Evidence Lower BOund}. …
Bayesian paradigm takes advantage of well fitting complicated survival models and feasible computing in survival analysis owing to the superiority in tackling the complex censoring scheme, compared with the frequentist paradigm. In this chapter, we aim to display the latest tendency in Bayesian computing, in the sense of automating the posterior sampling, through Bayesian analysis of survival modeling for multivariate survival outcomes with complicated data structure. Motivated by relaxing the strong assumption of proportionality and the restriction of a common baseline population, we propose …
Carnegie Mellon University（卡内基梅隆大学）；Microsoft Research（微软研究院）；Cornell University（康奈尔大学）；University of Toronto（多伦多大学）
Models which estimate main effects of individual variables alongside interaction effects have an identifiability challenge: effects can be freely moved between main effects and interaction effects without changing the model prediction. This is a critical problem for interpretability because it permits "contradictory" models to represent the same function. To solve this problem, we propose pure interaction effects: variance in the outcome which cannot be represented by any smaller subset of features. This definition has an equivalence with the Functional ANOVA decomposition. To compute this dec…
Most studies on inequality in infant and child mortality compare average mortality rates between large groups of births, for example, comparing births from different countries, income groups, ethnicities, or different times. These studies do not measure within-group disparities. The few studies that have measured within-group variability in infant and child mortality have used tools from the income inequality literature, such as Gini indices. We show that the latter are inappropriate for infant and child mortality. We develop novel tools that are appropriate for analyzing infant and child mort…
Responding to the current urgent need for low carbon emissions and high efficiency in manufacturing processes, the relationships between three different machining factors (depth of cut, feed rate, and spindle rate) on power consumption and surface finish (roughness) were analysed by applying a Bayesian seemingly unrelated regressions (SUR) model. For the analysis, an optimization criterion was established and minimized by using an optimization algorithm that combines evolutionary algorithm methods with a derivative-based (quasi-Newton) method to find the optimal conditions for energy consumpti…
Functional BART with Shape Priors: A Bayesian Tree Approach to Constrained Functional Regression
The University of Texas Health Science Center at Houston（德克萨斯大学健康科学中心休斯顿分校）；Beijing Technology and Business University（北京工商大学）；Beijing Normal-Hong Kong Baptist University（香港中文大学（深圳））
Graphical models, used to express conditional dependence between random variables observed at various nodes, are used extensively in many fields such as genetics, neuroscience, and social network analysis. While most current statistical methods for estimating graphical models focus on scalar data, there is interest in estimating analogous dependence structures when the data observed at each node are functional, such as signals or images. In this paper, we propose a fully Bayesian regularization scheme for estimating functional graphical models. We first consider a direct Bayesian analog of the…
Multivariate functional data arise in a wide range of applications. One fundamental task is to understand the causal relationships among these functional objects of interest, which has not yet been fully explored. In this article, we develop a novel Bayesian network model for multivariate functional data where the conditional independence and causal structure are both encoded by a directed acyclic graph. Specifically, we allow the functional objects to deviate from Gaussian process, which is adopted by most existing functional data analysis models. The more reasonable non-Gaussian assumption i…
Typical Bayesian inference requires parameter identification via likelihood parameterization, which has invited criticism for being less flexible than the Frequentist framework and subject to misspecification. Though misspecification may be avoided by functional parameter inference under a nonparametric model space, there does not exist a flexible Bayesian semiparametric model that would allow full control over the marginal prior over any general functional parameter. We present the technique of $θ$-augmentation which helps us manipulate nonparametric models into semiparametric ones that direc…
We develop a Bayesian graphical modeling framework for functional data for correlated multivariate random variables observed over a continuous domain. Our method leads to graphical Markov models for functional data which allows the graphs to vary over the functional domain. The model involves estimation of graphical models that evolve functionally in a nonparametric fashion while accounting for within-functional correlations and borrowing strength across functional positions so contiguous locations are encouraged but not forced to have similar graph structure and edge strength. We utilize a st…
