利用冗余、递归和重复:如何在十分钟内将数百万个重复与十行代码链接起来
Exploiting Redundancy, Recurrence and Parallelism: How to Link Millions of Addresses with Ten Lines of Code in Ten Minutes
摘要:
准确和有效的记录链接是澳大利亚政府机构面临的一个特别重要的挑战,他们认识到,所谓的邪恶社会问题最好通过建立在大规模数据融合基础上的伙伴关系来解决。名称和地址是最常见的属性,来自不同政府机构的数据可以在此基础上进行链接。在本文中,我们主要研究地址链接问题。当数据存在重大质量问题时,联系就特别成问题。处理质量问题最常见的方法是在链接之前对原始数据进行分析。然而,如果在标准化中犯了错误,通常不可能从中恢复以正确地执行链接。本文提出了一种新的地址链接算法,特别是实用的链接大型不同的地址集,是高度可扩展的,强大的数据质量问题和简单的实现。它避免了劳动密集型和有问题的地址标准化的需要。我们证明了该算法的有效性,通过匹配两个大的地址数据集,从两个政府机构具有良好的准确性和计算效率。
英文摘要:
Accurate and efficient record linkage is an open challenge of particular relevance to Australian Government Agencies, who recognise that so-called wicked social problems are best tackled by forming partnerships founded on large-scale data fusion. Names and addresses are the most common attributes on which data from different government agencies can be linked. In this paper, we focus on the problem of address linking. Linkage is particularly problematic when the data has significant quality issues. The most common approach for dealing with quality issues is to standardise raw data prior to linking. If a mistake is made in standardisation, however, it is usually impossible to recover from it to perform linkage correctly. This paper proposes a novel algorithm for address linking that is particularly practical for linking large disparate sets of addresses, being highly scalable, robust to data quality issues and simple to implement. It obviates the need for labour intensive and problematic address standardisation. We demonstrate the efficacy of the algorithm by matching two large address datasets from two government agencies with good accuracy and computational efficiency.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。