arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2003.08349cs.SE
收藏

从2B Git提交中提取的3800万个作者ID的数据集和身份解析方法

A Dataset and an Approach for Identity Resolution of 38 Million Author IDs extracted from 2B Git Commits

Tanner Fry, Tapajit Dey, Andrey Karnauch, Audris Mockus

摘要:

从开源项目中收集的数据提供了对大型软件生态系统建模的方法,但通常存在数据质量问题,特别是代码提交中的多个作者标识字符串实际上可能与一个开发人员相关联。虽然已经提出了许多方法来解决这个问题,但它们要么是需要手动调整的算法,要么需要太多的计算时间来对例如World of Code集合中的38M作者ID进行成对比较。在本文中,我们提出了一种方法,该方法可以在整个数据集中找到属于单个开发人员的所有作者ID,并共享被发现具有别名的所有作者ID的列表。为此,我们首先创建可能相关的作者ID块,然后使用机器学习模型来预测这些可能相关的ID中哪些属于同一个开发人员。我们处理了大约3800万个作者ID,发现大约1480万个ID有别名,属于540万个不同的开发人员,每个开发人员的别名中位数为2。该数据集可用于在整个OSS生态系统级别创建更准确的开发人员行为模型,并可用于提供快速解析新作者ID的服务。

英文摘要:

The data collected from open source projects provide means to model large software ecosystems, but often suffer from data quality issues, specifically, multiple author identification strings in code commits might actually be associated with one developer. While many methods have been proposed for addressing this problem, they are either heuristics requiring manual tweaking, or require too much calculation time to do pairwise comparisons for 38M author IDs in, for example, the World of Code collection. In this paper, we propose a method that finds all author IDs belonging to a single developer in this entire dataset, and share the list of all author IDs that were found to have aliases. To do this, we first create blocks of potentially connected author IDs and then use a machine learning model to predict which of these potentially related IDs belong to the same developer. We processed around 38 million author IDs and found around 14.8 million IDs to have an alias, which belong to 5.4 million different developers, with the median number of aliases being 2 per developer. This dataset can be used to create more accurate models of developer behaviour at the entire OSS ecosystem level and can be used to provide a service to rapidly resolve new author IDs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑