arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2409.18658cs.SE
收藏

SEART Data Hub:简化大规模源代码挖掘与预处理

SEART Data Hub: Streamlining Large-Scale Source Code Mining and Pre-Processing

Ozren Dabić, Rosalia Tufano, Gabriele Bavota

AI总结:

本文提出SEART Data Hub,一个简化大规模源代码挖掘与预处理的Web应用,允许研究人员通过简单界面指定挖掘和预处理条件,快速构建定制化数据集,显著降低时间和计算资源成本。

摘要:

大规模代码数据集在软件工程(SE)研究中扮演着越来越核心的角色。这是以下两点因素的结果:(i)挖掘软件存储库(MSR)社区的成功,推动了SE中实证研究的标准;以及(ii)近期深度学习(DL)在软件工程中的兴起,其模型在大型源代码数据集上进行训练和测试。尽管文献中存在一些即用型数据集,但研究人员通常需要构建并预处理自己的数据集,以满足他们正在研究的技术或研究的特定要求。这意味着在时间和计算资源方面存在巨大成本。在本研究中,我们提出了SEART Data Hub,这是一个Web应用程序,允许轻松构建和预处理包含从公共GitHub存储库挖掘的代码的大规模数据集。通过简单的Web界面,研究人员可以指定一组挖掘标准(例如,仅从拥有超过100名贡献者和超过1,000次提交的存储库中收集代码),以及他们想要执行的特定预处理步骤(例如,删除重复项、测试代码、存在语法错误的实例)。提交请求后,用户将在几个小时内收到一封包含所需数据集下载链接的电子邮件。展示SEART Data Hub的视频可在https://youtu.be/lCgQaA7CYWA获取。

英文摘要:

Large-scale code datasets have acquired an increasingly central role in software engineering (SE) research. This is the result of (i) the success of the mining software repositories (MSR) community, that pushed the standards of empirical studies in SE; and (ii) the recent advent of deep learning (DL) in software engineering, with models trained and tested on large source code datasets. While there exist some ready-to-use datasets in the literature, researchers often need to build and pre-process their own dataset to meet specific requirements of the study/technique they are working on. This implies a substantial cost in terms of time and computational resources. In this work we present the SEART Data Hub, a web application that allows to easily build and pre-process large-scale datasets featuring code mined from public GitHub repositories. Through a simple web interface, researchers can specify a set of mining criteria (e.g., only collect code from repositories having more than 100 contributors and more than 1,000 commits) as well as specific pre-processing steps they want to perform (e.g., remove duplicates, test code, instances with syntax errors). After submitting the request, the user will receive an email with a download link for the required dataset within a few hours. A video showcasing the SEART Data Hub is available at https://youtu.be/lCgQaA7CYWA.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑