arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1911.06348cs.SE
收藏

跨项目缺陷预测模型基于时间的结论稳定性

On the Time-Based Conclusion Stability of Cross-Project Defect Prediction Models

Abdul Ali Bangash, Hareem Sahar, Abram Hindle, Karim Ali

摘要:

经验软件工程的研究人员经常根据可观察的数据(如缺陷报告)提出索赔。不幸的是,在许多情况下,这些主张被泛化,超出了已经评估的数据集。对于相同的软件项目,研究人员的结论在一年后还能成立吗?也许不是。最近的研究表明,在软件分析领域,不同数据集的结论通常不一致。在这篇文章中,我们实证研究是否在该地区的缺陷预测的结论真正表现出稳定性在整个时间或没有。我们的研究采用了一种时间感知的评估方法,模型只在过去进行训练,评估只在未来执行。通过这种时间感知的评估,我们表明,根据我们评估缺陷预测器的时间段,它们的性能,在F分数,曲线下面积(AUC)和马修斯相关系数(MCC)方面,变化和它们的结果是不一致的。产品的下一个版本与之前的版本有很大的不同,可能会极大地改变缺陷预测性能。因此,在不知道结论稳定性的情况下,经验软件工程研究人员应该将他们对性能的声明限制在评估的上下文中,因为关于缺陷预测性能的广泛声明可能与分析中产品的下一个即将发布的版本相矛盾。

英文摘要:

Researchers in empirical software engineering often make claims based on observable data such as defect reports. Unfortunately, in many cases, these claims are generalized beyond the data sets that have been evaluated. Will the researcher's conclusions hold a year from now for the same software projects? Perhaps not. Recent studies show that in the area of Software Analytics, conclusions over different data sets are usually inconsistent. In this article, we empirically investigate whether conclusions in the area of defect prediction truly exhibit stability throughout time or not. Our investigation applies a time-aware evaluation approach where models are trained only on the past, and evaluations are executed only on the future. Through this time-aware evaluation, we show that depending on which time period we evaluate defect predictors, their performance, in terms of F-Score, the area under the curve (AUC), and Mathews Correlation Coefficient (MCC), varies and their results are not consistent. The next release of a product, which is significantly different from its prior release, may drastically change defect prediction performance. Therefore, without knowing about the conclusion stability, empirical software engineering researchers should limit their claims of performance within the contexts of evaluation, because broad claims about defect prediction performance might be contradicted by the next upcoming release of a product under analysis.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑