arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2302.00394cs.SE
收藏

缺陷预测模型的一致性性能评估

Toward a consistent performance evaluation for defect prediction models

Xutong Liu, Shiran Liu, Zhaoqiang Guo, Peng Zhag, Yibiao Yang, Huihui Liu, Hongmin Lu, Yanhui Li, Lin Chen, Yuming Zhou

摘要:

在缺陷预测领域,已经提出了许多缺陷预测模型,并且不断有新的模型被开发出来。然而,对于如何评估新提出的模型的性能,还没有达成共识。在本文中,我们的目标是提出物质,一个框架,以Ard一个consistentpErprisoncompaRison,这使得模型的性能直接比较不同的研究。我们采取三个行动来建立一个一致的评估框架,缺陷预测模型。首先,我们提出了一个简单易用的无监督基线模型ONE(glObal baseliNe modEl),以提供“单点比较”。其次,我们建议使用SQA-effort-aligned阈值设置来进行公平的比较。第三,我们建议统一报告评估结果,并为此提供一套核心绩效指标,以便进行跨研究比较,取得真正的进展。实验结果表明,MATTER可以作为一个有效的框架来支持对缺陷预测模型的一致性性能评估,从而可以帮助确定新提出的缺陷预测模型是否对实践者有用,并告知缺陷预测道路上的真正进展。此外,当应用MATTER来评估近年来提出的代表性缺陷预测模型时,我们发现其中大多数(如果不是全部)在SQA工作量感知预测性能方面并不优于简单的基线模型ONE。这表明缺陷预测的实际进展被高估了。因此,我们建议,在未来的研究中,当提出任何新的缺陷预测模型时,应使用MATTER来评估其实际有用性(在相同的基准测试数据集上),以推进缺陷预测的科学进步。

英文摘要:

In defect prediction community, many defect prediction models have been proposed and indeed more new models are continuously being developed. However, there is no consensus on how to evaluate the performance of a newly proposed model. In this paper, we aim to propose MATTER, a fraMework towArd a consisTenT pErformance compaRison, which makes model performance directly comparable across different studies. We take three actions to build a consistent evaluation framework for defect prediction models. First, we propose a simple and easy-to-use unsupervised baseline model ONE (glObal baseliNe modEl) to provide "a single point of comparison". Second, we propose using the SQA-effort-aligned threshold setting to make a fair comparison. Third, we suggest reporting the evaluation results in a unified way and provide a set of core performance indicators for this purpose, thus enabling an across-study comparison to attain real progress. The experimental results show that MATTER can serve as an effective framework to support a consistent performance evaluation for defect prediction models and hence can help determine whether a newly proposed defect prediction model is practically useful for practitioners and inform the real progress in the road of defect prediction. Furthermore, when applying MATTER to evaluate the representative defect prediction models proposed in recent years, we find that most of them (if not all) are not superior to the simple baseline model ONE in terms of the SQA-effort awareness prediction performance. This reveals that the real progress in defect prediction has been overestimated. We hence recommend that, in future studies, when any new defect prediction model is proposed, MATTER should be used to evaluate its actual usefulness (on the same benchmark test data sets) to advance scientific progress in defect prediction.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑