arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1809.01604cs.LGcs.AIstat.ML
收藏

通过深度学习合并数据集

Merging datasets through deep learning

IBM Research(IBM研究院);Yeshiva University(叶史瓦大学)

Kavitha Srinivas, Abraham Gale, Julian Dolby

摘要:

合并数据集是数据分析的关键操作。合并的一个常见要求是连接同一实体具有不同表面形式的列(例如,一个人的名字可以表示为“Douglas Adams”或“Adams,Douglas”)。类似地,本体对齐可能需要识别相同实体的不同表面形式,特别是当本体独立开发时。然而,数据管理系统当前限于基于字符串相等或至多使用字符串相似性来执行合并。我们提出了一种基于深度学习模型执行合并的方法。我们的方法依赖于(a)创建一个深度学习模型,将实体的表面形式映射到一组向量中,使得同一实体的替代形式在向量空间中最接近,(b)使用最近邻算法对这些向量进行索引,以找到可能连接在一起的形式。为了建立这些模型,我们必须适应技术从度量学习由于数据的特点,具体来说,我们描述了新的样本选择技术和损失函数,为这个问题的工作。为了评估我们的方法,我们使用维基数据作为基础事实,并从大约110万人的名字(20万个身份)和13万公司名称(7万个身份)的数据集构建模型。我们开发的模型允许连接的精度为0.75 - 0.81,召回率为0.74 - 0.81。我们使模型可用于在多个数据集中对齐人员或公司。

英文摘要:

Merging datasets is a key operation for data analytics. A frequent requirement for merging is joining across columns that have different surface forms for the same entity (e.g., the name of a person might be represented as "Douglas Adams" or "Adams, Douglas"). Similarly, ontology alignment can require recognizing distinct surface forms of the same entity, especially when ontologies are independently developed. However, data management systems are currently limited to performing merges based on string equality, or at best using string similarity. We propose an approach to performing merges based on deep learning models. Our approach depends on (a) creating a deep learning model that maps surface forms of an entity into a set of vectors such that alternate forms for the same entity are closest in vector space, (b) indexing these vectors using a nearest neighbors algorithm to find the forms that can be potentially joined together. To build these models, we had to adapt techniques from metric learning due to the characteristics of the data; specifically we describe novel sample selection techniques and loss functions that work for this problem. To evaluate our approach, we used Wikidata as ground truth and built models from datasets with approximately 1.1M people's names (200K identities) and 130K company names (70K identities). We developed models that allow for joins with precision@1 of .75-.81 and recall of .74-.81. We make the models available for aligning people or companies across multiple datasets.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑