arXivDaily arXiv每日学术速递 周一至周五更新
北美退伍军人节(Veterans Day),12 日无更新
arXiv 1609.06265cs.AIcs.DB
收藏

大规模稀疏数据集实体解析的包围分块方案

An Ensemble Blocking Scheme for Entity Resolution of Large and Sparse Datasets

CareerBuilderLLC(凯业必达有限责任公司);The University of Texas at Austin(德克萨斯大学奥斯汀分校)

Janani Balaji, Faizan Javed, Mayank Kejriwal, Chris Min, Sam Sander, Ozgur Ozturk

摘要:

实体解析(Entity Resolution),也称为记录链接或重复数据删除,是指识别同一实体的重复版本并将其合并为统一表示的过程。标准做法是使用基于规则或基于机器学习的模型,该模型比较实体对并分配分数以表示对的匹配/不匹配状态。然而,在所有记录对上执行详尽的逐对比较导致二次匹配器复杂性,因此在匹配之前执行分块步骤以将相似实体分组为较小的块,然后匹配器可以详尽地检查。已经开发了几种分块方案,以高效地将输入数据集分块到可管理的组中。在凯业必达(CB),我们对从不同来源收集的具有不同信息内容的人员配置文件的大规模数据集执行重复数据删除。我们观察到,由于我们的数据源的多方面性质,采用单一的阻塞技术并不能覆盖所有可能的场景。在本文中,我们描述了我们的集成方法来阻止,结合两种不同的阻塞技术,以利用各自的优势。

英文摘要:

Entity Resolution, also called record linkage or deduplication, refers to the process of identifying and merging duplicate versions of the same entity into a unified representation. The standard practice is to use a Rule based or Machine Learning based model that compares entity pairs and assigns a score to represent the pairs' Match/Non-Match status. However, performing an exhaustive pair-wise comparison on all pairs of records leads to quadratic matcher complexity and hence a Blocking step is performed before the Matching to group similar entities into smaller blocks that the matcher can then examine exhaustively. Several blocking schemes have been developed to efficiently and effectively block the input dataset into manageable groups. At CareerBuilder (CB), we perform deduplication on massive datasets of people profiles collected from disparate sources with varying informational content. We observed that, employing a single blocking technique did not cover the base for all possible scenarios due to the multi-faceted nature of our data sources. In this paper, we describe our ensemble approach to blocking that combines two different blocking techniques to leverage their respective strengths.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑