arXiv 1410.47922014-10-17stat.MEstat.ML

Variational Bayes for Merging Noisy Databases

Tamara Broderick, Rebecca C. Steorts

备注：: 12 pages

英文摘要：

Bayesian entity resolution merges together multiple, noisy databases and returns the minimal collection of unique individuals represented, together with their true, latent record values. Bayesian methods allow flexible generative models that share power across databases as well as principled quantification of uncertainty for queries of the final, resolved database. However, existing Bayesian methods for entity resolution use Markov monte Carlo method (MCMC) approximations and are too slow to run on modern databases containing millions or billions of records. Instead, we propose applying variational approximations to allow scalable Bayesian inference in these models. We derive a coordinate-ascent approximation for mean-field variational Bayes, qualitatively compare our algorithm to existing methods, note unique challenges for inference that arise from the expected distribution of cluster sizes in entity resolution, and discuss directions for future work in this domain.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

阅读原文

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

本文内容

英文摘要相关论文

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

继续探索

相关论文

Bayesian inference of chaotic dynamics by merging data assimilation, machine learning and expectation-maximization

Nansen Environmental and Remote Sensing Center（南森环境与遥感中心）；Sorbonne University（索邦大学）；University of Reading（雷丁大学）；Utrecht University（乌得勒支大学）

The reconstruction from observations of high-dimensional chaotic dynamics such as geophysical flows is hampered by (i) the partial and noisy observations that can realistically be obtained, (ii) the need to learn from long time series of data, and (iii) the unstable nature of the dynamics. To achieve such inference from the observations over long time series, it has been suggested to combine data assimilation and machine learning in several ways. We show how to unify these approaches from a Bayesian perspective using expectation-maximization and coordinate descents. In doing so, the model, the…

2020-03-27cs.LGphysics.ao-phstat.ML

泊松复合决策的贝叶斯与拟贝叶斯经验贝叶斯方法的融合

Merging of Bayes and quasi-Bayes empirical Bayes procedures for Poisson compound decisions

University of Torino（都灵大学）；Collegio Carlo Alberto（卡洛·阿尔贝托学院）；Bocconi University（博科尼大学）

研究泊松复合决策问题中贝叶斯与拟贝叶斯经验贝叶斯策略的融合，建立边际概率质量函数的集中速率并证明频率学派融合结果，数值实验显示拟贝叶斯策略在保持精度的同时大幅降低计算成本。

2026-07-07math.STstat.MEstat.OT

A Bayesian model for microarray datasets merging

The aggregation of microarray datasets originating from different studies is still a difficult open problem. Currently, best results are generally obtained by the so-called meta-analysis approach, which aggregates results from individual datasets, instead of analyzing aggre-gated datasets. In order to tackle such aggregation problems, it is necessary to correct for interstudy variability prior to aggregation. The goal of this paper is to present a new approach for microarray datasets merging, based upon explicit modeling of interstudy variability and gene variability. We develop and demonstrat…

2015-10-27q-bio.QMstat.ME

联合贝叶斯模型用于验证空间健康事件数据库与金标准：分离全局和局部差异

Joint Bayesian models for validating spatial health-event databases against a gold standard: separating global and local discrepancies

提出贝叶斯框架，通过误差模型（随机和结构化）与共享成分模型，从全局和局部两个维度验证空间健康事件数据库与金标准的一致性。

2026-05-22stat.ME

Bayesian Estimation of Bipartite Matchings for Record Linkage

The bipartite record linkage task consists of merging two disparate datafiles containing information on two overlapping sets of entities. This is non-trivial in the absence of unique identifiers and it is important for a wide variety of applications given that it needs to be solved whenever we have to combine information from different sources. Most statistical techniques currently used for record linkage are derived from a seminal paper by Fellegi and Sunter (1969). These techniques usually assume independence in the matching statuses of record pairs to derive estimation procedures and optima…

2016-01-25stat.APstat.MEstat.ML

二元结局与时间-事件结局贝叶斯元分析的经验先验分布

Empirical prior distributions for Bayesian meta-analyses of binary and time to event outcomes

该研究利用Cochrane数据库开发二元和时间-事件结局贝叶斯元分析的经验先验分布，发现二元结局多无效应或异质性，时间-事件结局则相反，并给出具体分布及R和JASP应用示例。

2023-06-20stat.ME

更多相关论文收起更多论文

RMCMC: A System for Updating Bayesian Models

A system to update estimates from a sequence of probability distributions is presented. The aim of the system is to quickly produce estimates with a user-specified bound on the Monte Carlo error. The estimates are based upon weighted samples stored in a database. The stored samples are maintained such that the accuracy of the estimates and quality of the samples is satisfactory. This maintenance involves varying the number of samples in the database and updating their weights. New samples are generated, when required, by a Markov chain Monte Carlo algorithm. The system is demonstrated using a …

2014-06-03stat.ME

Bayesian Pseudo Posterior Mechanism under Asymptotic Differential Privacy

We propose a Bayesian pseudo posterior mechanism to generate record-level synthetic databases equipped with an $(ε,δ)-$ probabilistic differential privacy (pDP) guarantee, where $δ$ denotes the probability that any observed database exceeds $ε$. The pseudo posterior mechanism employs a data record-indexed, risk-based weight vector with weight values $\in [0, 1]$ that surgically downweight the likelihood contributions for high-risk records for model estimation and the generation of record-level synthetic data for public release. The pseudo posterior synthesizer constructs a weight for each data…

2021-08-13stat.ME

Entity Resolution with Empirically Motivated Priors

Databases often contain corrupted, degraded, and noisy data with duplicate entries across and within each database. Such problems arise in citations, medical databases, genetics, human rights databases, and a variety of other applied settings. The target of statistical inference can be viewed as an unsupervised problem of determining the edges of a bipartite graph that links the observed records to unobserved latent entities. Bayesian approaches provide attractive benefits, naturally providing uncertainty quantification via posterior probabilities. We propose a novel record linkage approach ba…

2015-04-27stat.ME

Probabilistic Record Linkage and Deduplication after Indexing, Blocking, and Filtering

Probabilistic record linkage, the task of merging two or more databases in the absence of a unique identifier, is a perennial and challenging problem. It is closely related to the problem of deduplicating a single database, which can be cast as linking a single database against itself. In both cases the number of possible links grows rapidly in the size of the databases under consideration, and in most applications it is necessary to first reduce the number of record pairs that will be compared. Spurred by practical considerations, a range of methods have been developed for this task. These me…

2016-03-25stat.ME

Generalized Bayesian Record Linkage and Regression with Exact Error Propagation

Record linkage (de-duplication or entity resolution) is the process of merging noisy databases to remove duplicate entities. While record linkage removes duplicate entities from such databases, the downstream task is any inferential, predictive, or post-linkage task on the linked data. One goal of the downstream task is obtaining a larger reference data set, allowing one to perform more accurate statistical analyses. In addition, there is inherent record linkage uncertainty passed to the downstream task. Motivated by the above, we propose a generalized Bayesian record linkage method and consid…

2018-10-11stat.ME

Practical Bayesian Inference for Record Linkage

Probabilistic record linkage (PRL) is the process of determining which records in two databases correspond to the same underlying entity in the absence of a unique identifier. Bayesian solutions to this problem provide a powerful mechanism for propagating uncertainty due to uncertain links between records (via the posterior distribution). However, computational considerations severely limit the practical applicability of existing Bayesian approaches. We propose a new computational approach, providing both a fast algorithm for deriving point estimates of the linkage structure that properly acco…

2017-12-01stat.ME

Incorporating prior information and borrowing information in high-dimensional sparse regression using the horseshoe and variational Bayes

We introduce a sparse high-dimensional regression approach that can incorporate prior information on the regression parameters and can borrow information across a set of similar datasets. Prior information may for instance come from previous studies or genomic databases, and information borrowed across a set of genes or genomic networks. The approach is based on prior modelling of the regression parameters using the horseshoe prior, with a prior on the sparsity index that depends on external information. Multiple datasets are integrated by applying an empirical Bayes strategy on hyperparameter…

2019-01-29stat.ME