在使用或验证缺陷预测模型时需要删除最后发布的数据
On the Need of Removing Last Releases of Data When Using or Validating Defect Prediction Models
摘要：
为了开发和训练缺陷预测模型，研究人员依赖于其中缺陷归因于人工制品的数据集，例如，一个给定的释放类。然而，这种数据集的创建远非完美。可能会发生这样的情况：一个缺陷在引入几个版本后才被发现：这种现象被称为“休眠缺陷”。这意味着，如果我们今天观察一个类在其当前版本中的状态，它可以被认为是无缺陷的，而事实并非如此。我们称之为“打鼾”的噪音组成的这类，受休眠缺陷只。我们推测打鼾的存在会对分类器的准确性和评估产生负面影响。此外，较早的版本可能包含比旧版本更多的打鼾类，因此，从数据集中删除最新的版本可以减少打鼾影响并提高分类器的准确性。在本文中，我们调查的打鼾噪声对分类器的准确性和他们的评估，以及有效性的一种可能的对策，包括在删除最后发布的数据的影响。我们分析了15个机器学习缺陷预测分类器的准确性，这些分类器的数据来自Apache生态系统的19个开源项目的4,000多个bug和600个版本。我们的研究结果表明，平均而言，在项目中：（i）打鼾的存在降低了缺陷预测分类器的召回率;（ii）受打鼾影响的评估可能无法识别最佳分类器，以及（iii）从最近发布的数据中删除有助于显着提高分类器的准确性。总之，本文提供了关于如何通过减轻打鼾的影响来创建软件缺陷数据集的见解。
英文摘要：
To develop and train defect prediction models, researchers rely on datasets in which a defect is attributed to an artifact, e.g., a class of a given release. However, the creation of such datasets is far from being perfect. It can happen that a defect is discovered several releases after its introduction: this phenomenon has been called "dormant defects". This means that, if we observe today the status of a class in its current version, it can be considered as defect-free while this is not the case. We call "snoring" the noise consisting of such classes, affected by dormant defects only. We conjecture that the presence of snoring negatively impacts the classifiers' accuracy and their evaluation. Moreover, earlier releases likely contain more snoring classes than older releases, thus, removing the most recent releases from a dataset could reduce the snoring effect and improve the accuracy of classifiers. In this paper we investigate the impact of the snoring noise on classifiers' accuracy and their evaluation, and the effectiveness of a possible countermeasure consisting in removing the last releases of data. We analyze the accuracy of 15 machine learning defect prediction classifiers on data from more than 4,000 bugs and 600 releases of 19 open source projects from the Apache ecosystem. Our results show that, on average across projects: (i) the presence of snoring decreases the recall of defect prediction classifiers; (ii) evaluations affected by snoring are likely unable to identify the best classifiers, and (iii) removing data from recent releases helps to significantly improve the accuracy of the classifiers. On summary, this paper provides insights on how to create a software defect dataset by mitigating the effect of snoring.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
