In this paper, we study the accuracy of values aggregated over classes predicted by a classification algorithm. The problem is that the resulting aggregates (e.g., sums of a variable) are known to be biased. The bias can be large even for highly accurate classification algorithms, in particular when dealing with class-imbalanced data. To correct this bias, the algorithm's classification error rates have to be estimated. In this estimation, two issues arise when applying existing bias correction methods. First, inaccuracies in estimating classification error rates have to be taken into account. Second, impermissible estimates, such as a negative estimate for a positive value, have to be dismissed. We show that both issues are relevant in applications where the true labels are known only for a small set of data points. We propose a novel bias correction method using Bayesian inference. The novelty of our method is that it imposes constraints on the model parameters. We show that our method solves the problem of biased classification-based aggregates as well as the two issues above, in the general setting of multi-class classification. In the empirical evaluation, using a binary classifier on a real-world dataset of company tax returns, we show that our method outperforms existing methods in terms of mean squared error.
The present work describes simulation studies to compare the performances of bayesian wavelet shrinkage methods in estimating component curves from aggregated functional data. To do so, five methods were considered: the bayesian shrinkage rule under logistic prior by Sousa (2020), bayesian shrinkage rule under beta prior by Sousa et al. (2020), Large Posterior Mode method by Cutillo et al. (2008), Amplitude-scale invariant Bayes Estimator by Figueiredo and Nowak (2001) and Bayesian Adaptive Multiresolution Smoother by Vidakovic and Ruggeri (2001). Further, the so called Donoho-Johnstone test f…
Growth curves are commonly used in modeling aimed at crop yield prediction. Fitting such curves often depends on availability of detailed observations, such as individual grape bunch weight or individual apple weight. However, in practice, aggregated weights (such as a bucket of grape bunches or apples) are available instead. While treating such bucket averages as if they were individual observations is tempting, it may introduce bias particularly with respect to population variance. In this paper we provide an elegant solution which enables estimation of individual weights using Dirichlet pri…
Bayesian random-effects meta-analysis of aggregate data on clinical events
University Medical Center Göttingen（哥廷根大学医学中心）；University of Marburg（马尔堡大学）；Merck KGaA（默克KGaA）；DZHK (German Center for Cardiovascular Research)（德国心血管研究中心）
We study the problem of separating a mixture of distributions, all of which come from interventions on a known causal bayesian network. Given oracle access to marginals of all distributions resulting from interventions on the network, and estimates of marginals from the mixture distribution, we want to recover the mixing proportions of different mixture components. We show that in the worst case, mixing proportions cannot be identified using marginals only. If exact marginals of the mixture distribution were known, under a simple assumption of excluding a few distributions from the mixture, we…
We study the problem of aggregation under the squared loss in the model of regression with deterministic design. We obtain sharp PAC-Bayesian risk bounds for aggregates defined via exponential weights, under general assumptions on the distribution of errors and on the functions to aggregate. We then apply these results to derive sparsity oracle inequalities.
Surrogate endpoints play an important role in drug development when they can be used to measure treatment effect early compared to the final clinical outcome and to predict clinical benefit or harm. Such endpoints are assessed for their predictive value of clinical benefit by investigating the surrogate relationship between treatment effects on the surrogate and final outcomes using meta-analytic methods. When surrogate relationships vary across treatment classes, such validation may fail due to limited data within each treatment class. In this paper, two alternative Bayesian meta-analytic met…
We develop a set of scalable Bayesian inference procedures for a general class of nonparametric regression models. Specifically, nonparametric Bayesian inferences are separately performed on each subset randomly split from a massive dataset, and then the obtained local results are aggregated into global counterparts. This aggregation step is explicit without involving any additional computation cost. By a careful partition, we show that our aggregated inference results obtain an oracle rule in the sense that they are equivalent to those obtained directly from the entire data (which are computa…
Managing risk at the aggregate level is crucial for banks and financial institutions as required by the Basel III framework. In this paper, we introduce discrete time Bayesian state space models with Poisson measurements to model aggregate mortgage default rate. We discuss parameter updating, filtering, smoothing, forecasting and estimation using Markov chain Monte Carlo methods. In addition, we investigate the dynamic behavior of the default rate and the effects of macroeconomic variables. We illustrate the use of the proposed models using actual U.S. residential mortgage data and discuss ins…
This paper applies the Bayesian Model Averaging (BMA) statistical ensemble technique to estimate small molecule solvation free energies. There is a wide range of methods available for predicting solvation free energies, ranging from empirical statistical models to ab initio quantum mechanical approaches. Each of these methods is based on a set of conceptual assumptions that can affect predictive accuracy and transferability. Using an iterative statistical process, we have selected and combined solvation energy estimates using an ensemble of 17 diverse methods from the fourth Statistical Assess…
Tufts University（塔夫茨大学）；Tufts Medical Center（塔夫茨医疗中心）
We address the problem of modeling constrained hospital resources in the midst of the COVID-19 pandemic in order to inform decision-makers of future demand and assess the societal value of possible interventions. For broad applicability, we focus on the common yet challenging scenario where patient-level data for a region of interest are not available. Instead, given daily admissions counts, we model aggregated counts of observed resource use, such as the number of patients in the general ward, in the intensive care unit, or on a ventilator. In order to explain how individual patient trajector…
Many organizations face critical decisions that rely on forecasts of binary events. In these situations, organizations often gather forecasts from multiple experts or models and average those forecasts to produce a single aggregate forecast. Because the average forecast is known to be underconfident, methods have been proposed that create an aggregate forecast more extreme than the average forecast. But is it always appropriate to extremize the average forecast? And if not, when is it appropriate to anti-extremize (i.e., to make the aggregate forecast less extreme)? To answer these questions, …
Bivariate meta-analysis provides a useful framework for combining information across related studies and has been utilised to combine evidence from clinical studies to evaluate treatment efficacy on two outcomes. It has also been used to investigate surrogacy patterns between treatment effects on the surrogate endpoint and the final outcome. Surrogate endpoints play an important role in drug development when they can be used to measure treatment effect early compared to the final outcome and to predict clinical benefit or harm. The standard bivariate meta-analytic approach models the observed …
A general challenge in statistics is prediction in the presence of multiple candidate models or learning algorithms. Model aggregation tries to combine all predictive distributions from individual models, which is more stable and flexible than single model selection. In this article we describe when and how to aggregate models under the lens of Bayesian decision theory. Among two widely used methods, Bayesian model averaging (BMA) and Bayesian stacking, we compare their predictive performance, and review their theoretical optimality, probabilistic interpretation, practical implementation, and …
In order to investigate the breast cancer prediction problem on the aging population with the grades of DCIS, we conduct a tree augmented naive Bayesian network experiment trained and tested on a large clinical dataset including consecutive diagnostic mammography examinations, consequent biopsy outcomes and related cancer registry records in the population of women across all ages. The aggregated results of our ten-fold cross validation method recommend a biopsy threshold higher than 2% for the aging population.
Due to increased awareness of data protection and corresponding laws many data, especially involving sensitive personal information, are not publicly accessible. Accordingly, many data collecting agencies only release aggregated data, e.g. providing the mean and selected quantiles of population distributions. Yet, research and scientific understanding, e.g. for medical diagnostics or policy advice, often relies on data access. To overcome this tension, we propose a Bayesian method for learning from quantile information. Being based on order statistics of finite samples our method adequately an…
Age-disaggregated health data is crucial for effective public health planning and monitoring. Monitoring under-five mortality, for example, requires highly detailed age data since the distribution of potential causes of death varies substantially within the first few years of life. Comparative researchers often have to rely on multiple data sources yet, these sources often have ages aggregated at different levels, making it difficult to combine the data into a single, coherent picture. To address this challenge in the context of under-five cause-specific mortality, we propose a Bayesian approa…
We extend the standard Bayesian multivariate Gaussian generative data classifier by considering a generalization of the conjugate, normal-Wishart prior distribution and by deriving the hyperparameters analytically via evidence maximization. The behaviour of the optimal hyperparameters is explored in the high-dimensional data regime. The classification accuracy of the resulting generalized model is competitive with state-of-the art Bayesian discriminant analysis methods, but without the usual computational burden of cross-validation.
University of Waterloo（滑铁卢大学）；Vector Institute（向量研究所）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）
Making predictions robust is an important challenge. A separate challenge in federated learning (FL) is to reduce the number of communication rounds, particularly since doing so reduces performance in heterogeneous data settings. To tackle both issues, we take a Bayesian perspective on the problem of learning a global model. We show how the global predictive posterior can be approximated using client predictive posteriors. This is unlike other works which aggregate the local model space posteriors into the global model space posterior, and are susceptible to high approximation errors due to th…
Varying coefficient models (VCMs) are widely used for estimating nonlinear regression functions for functional data. Their Bayesian variants using Gaussian process priors on the functional coefficients, however, have received limited attention in massive data applications, mainly due to the prohibitively slow posterior computations using Markov chain Monte Carlo (MCMC) algorithms. We address this problem using a divide-and-conquer Bayesian approach. We first create a large number of data subsamples with much smaller sizes. Then, we formulate the VCM as a linear mixed-effects model and develop …
University of Chicago（芝加哥大学）；University of Toronto（多伦多大学）；Vector Institute（矢量研究所）
While uncertainty estimation is a well-studied topic in deep learning, most such work focuses on marginal uncertainty estimates, i.e. the predictive mean and variance at individual input locations. But it is often more useful to estimate predictive correlations between the function values at different input locations. In this paper, we consider the problem of benchmarking how accurately Bayesian models can estimate predictive correlations. We first consider a downstream task which depends on posterior predictive correlations: transductive active learning (TAL). We find that TAL makes better us…
Methods for reasoning under uncertainty are a key building block of accurate and reliable machine learning systems. Bayesian methods provide a general framework to quantify uncertainty. However, because of model misspecification and the use of approximate inference, Bayesian uncertainty estimates are often inaccurate -- for example, a 90% credible interval may not contain the true outcome 90% of the time. Here, we propose a simple procedure for calibrating any regression algorithm; when applied to Bayesian and probabilistic models, it is guaranteed to produce calibrated uncertainty estimates g…
Understanding the effects of social determinants of health on health outcomes requires data on characteristics of the neighborhoods in which subjects live. However, estimates of these characteristics are often aggregated over space and time in a fashion that diminishes their utility. Take, for example, estimates from the American Community Survey (ACS), a multi-year nationwide survey administered by the U.S. Census Bureau: estimates for small municipal areas are aggregated over 5-year periods, whereas 1-year estimates are only available for municipal areas with populations $>$65,000. Researche…
This paper introduces a novel parameter estimation method for the probability tables of Bayesian network classifiers (BNCs), using hierarchical Dirichlet processes (HDPs). The main result of this paper is to show that improved parameter estimation allows BNCs to outperform leading learning methods such as Random Forest for both 0-1 loss and RMSE, albeit just on categorical datasets. As data assets become larger, entering the hyped world of "big", efficient accurate classification requires three main elements: (1) classifiers with low-bias that can capture the fine-detail of large datasets (2) …
Feature selection represents a measure to reduce the complexity of high-dimensional datasets and gain insights into the systematic variation in the data. This aspect is of specific importance in domains that rely on model interpretability, such as life sciences. We propose UBayFS, an ensemble feature selection technique embedded in a Bayesian statistical framework. Our approach considers two sources of information: data and domain knowledge. We build a meta-model from an ensemble of elementary feature selectors and aggregate this information in a multinomial likelihood. The user guides UBayFS …
Microsoft Research（微软研究院）；University of California（加利福尼亚大学）
We discuss Bayesian methods for learning Bayesian networks when data sets are incomplete. In particular, we examine asymptotic approximations for the marginal likelihood of incomplete data given a Bayesian network. We consider the Laplace approximation and the less accurate but more efficient BIC/MDL approximation. We also consider approximations proposed by Draper (1993) and Cheeseman and Stutz (1995). These approximations are as efficient as BIC/MDL, but their accuracy has not been studied in any depth. We compare the accuracy of these approximations under the assumption that the Laplace app…
With the recent advances in electrical engineering, devices attached to free-ranging marine mammals today can collect oceanographic data in remarkable high spatial-temporal resolution. However, those data cannot be fully utilized without a matching high-resolution and accurate path of the animal, which is currently missing in this field. In this paper, we develop a Bayesian melding approach based on a Brownian Bridge process to combine the fine-resolution but seriously biased Dead-Reckoned path and the precise but sparse GPS measurements, which results in an accurate and high-resolution estima…
The aim of this paper is to provide some theoretical understanding of quasi-Bayesian aggregation methods non-negative matrix factorization. We derive an oracle inequality for an aggregated estimator. This result holds for a very general class of prior distributions and shows how the prior affects the rate of convergence.
Bayesian hierarchical models are increasingly popular for realistic modelling and analysis of complex data. This trend is accompanied by the need for flexible, general, and computationally efficient methods for model criticism and conflict detection. Usually, a Bayesian hierarchical model incorporates a grouping of the individual data points, for example individuals in repeated measurement data. In such cases, the following question arises: Are any of the groups "outliers", or in conflict with the remaining groups? Existing general approaches aiming to answer such questions tend to be extremel…
Federated learning aims to collaboratively train a strong global model by accessing users' locally trained models but not their own data. A crucial step is therefore to aggregate local models into a global model, which has been shown challenging when users have non-i.i.d. data. In this paper, we propose a novel aggregation algorithm named FedBE, which takes a Bayesian inference perspective by sampling higher-quality global models and combining them via Bayesian model Ensemble, leading to much robust aggregation. We show that an effective model distribution can be constructed by simply fitting …
Estimating global pairwise interaction effects, i.e., the difference between the joint effect and the sum of marginal effects of two input features, with uncertainty properly quantified, is centrally important in science applications. We propose a non-parametric probabilistic method for detecting interaction effects of unknown form. First, the relationship between the features and the output is modelled using a Bayesian neural network, capable of representing complex interactions and principled uncertainty. Second, interaction effects and their uncertainty are estimated from the trained model.…
Online social networks (OSN) contain extensive amount of information about the underlying society that is yet to be explored. One of the most feasible technique to fetch information from OSN, crawling through Application Programming Interface (API) requests, poses serious concerns over the the guarantees of the estimates. In this work, we focus on making reliable statistical inference with limited API crawls. Based on regenerative properties of the random walks, we propose an unbiased estimator for the aggregated sum of functions over edges and proved the connection between variance of the est…
Bayesian Optimization is a sample-efficient black-box optimization procedure that is typically applied to problems with a small number of independent objectives. However, in practice we often wish to optimize objectives defined over many correlated outcomes (or "tasks"). For example, scientists may want to optimize the coverage of a cell tower network across a dense grid of locations. Similarly, engineers may seek to balance the performance of a robot across dozens of different environments via constrained or robust optimization. However, the Gaussian Process (GP) models typically used as prob…
Data in the form of ranking lists are frequently encountered, and combining ranking results from different sources can potentially generate a better ranking list and help understand behaviors of the rankers. Of interest here are the rank data under the following settings: (i) covariate information available for the ranked entities; (ii) rankers of varying qualities or having different opinions; and (iii) incomplete ranking lists for non-overlapping subgroups. We review some key ideas built around the Thurstone model family by researchers in the past few decades and provide a unifying approach …
Sensitivity Analysis is a framework to assess how conclusions drawn from missing outcome data may be vulnerable to departures from untestable underlying assumptions. We extend the E-value, a popular metric for quantifying robustness of causal conclusions, to the setting of missing outcomes. With motivating examples from partially-observed Facebook conversion events, we present methodology for conducting Sensitivity Analysis at scale with three contributions. First, we develop a method for the Bayesian estimation of sensitivity parameters leveraging noisy benchmarks(e.g., aggregated reports for…
