arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2204.09165cs.SE
收藏

测试套件有效性度量评估:我们知道什么,我们应该做什么?

Test suite effectiveness metric evaluation: what do we know and what should we do?

Peng Zhang, Yang Wang, Xutong Liu, Yibiao Yang, Yanhui Li, Lin Chen, Ziyuan Wang, Chang-ai Sun, Yuming Zhou

摘要:

测试集有效性度量的比较一直是研究的热点。然而,以前的研究有不同的结论,甚至相互矛盾的比较不同的测试套件的有效性指标。我们发现最困扰我们社区的问题是,研究人员倾向于过度简化他们所使用的基本事实的描述。例如,一个常见的表达是“我们研究了真实故障和要评估的度量(MTE)之间的相关性”。然而,“真正的错误”的含义并不明确。因此,有必要仔细研究“真正的过失”的含义。如果没有这一点,它将对结论一知半解。为了应对这一挑战,我们提出了一个框架ASSENT(评估测试套件竞争力Ness meTrics),以指导后续研究。本质上,ASSENT由三个基本组件组成:地面实况,基准测试套件和协议指示器。首先,具体化地面真相,以确定测试套件之间的有效性的真实顺序。其次,生成一组基准测试集,并推导出它们的有效性的地面真值顺序。第三,对于基准测试套件,根据评估指标(MTE)生成有效性MTE顺序。最后,计算两个订单之间的一致性指标。在ASSENT下,我们能够比较不同测试套件有效性度量的准确性。我们应用ASSENT评估代表性的测试套件的有效性指标,包括突变评分指标和代码覆盖率指标。我们的研究结果表明,基于真实的故障,突变分数和包容突变分数是最好的度量,以量化测试套件的有效性。同时,通过使用突变体而不是真正的故障,MTEs将被高估超过20%的值。

英文摘要:

Comparing test suite effectiveness metrics has always been a research hotspot. However, prior studies have different conclusions or even contradict each other for comparing different test suite effectiveness metrics. The problem we found most troubling to our community is that researchers tend to oversimplify the description of the ground truth they use. For example, a common expression is that "we studied the correlation between real faults and the metric to evaluate (MTE)". However, the meaning of "real faults" is not clear-cut. As a result, there is a need to scrutinize the meaning of "real faults". Without this, it will be half-knowledgeable with the conclusions. To tackle this challenge, we propose a framework ASSENT (evAluating teSt Suite EffectiveNess meTrics) to guide the follow-up research. In nature, ASSENT consists of three fundamental components: ground truth, benchmark test suites, and agreement indicator. First, materialize the ground truth for determining the real order in effectiveness among test suites. Second, generate a set of benchmark test suites and derive their ground truth order in effectiveness. Third, for the benchmark test suites, generate the MTE order in effectiveness by the metric to evaluate (MTE). Finally, calculate the agreement indicator between the two orders. Under ASSENT, we are able to compare the accuracy of different test suite effectiveness metrics. We apply ASSENT to evaluate representative test suite effectiveness metrics, including mutation score metrics and code coverage metrics. Our results show that, based on the real faults, mutation score and subsuming mutation score are the best metrics to quantify test suite effectiveness. Meanwhile, by using mutants instead of real faults, MTEs will be overestimated by more than 20% in values.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑