arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2011.02861cs.SE
收藏

软件工程中复制结果的比较

Comparing the Results of Replications in Software Engineering

Adrian Santos, Sira Vegas, Markku Oivo, Natalia Juristo

摘要:

背景:有人认为,软件工程复制对于验证以前的实验结果是有用的。然而,尚未就如何检查结果是否在重复中保持一致达成一致。此外,一些作者认为,重复不验证以前的实验结果可以用来识别上下文变量造成的差异。目的:研究如何评估SE重复结果的(不)相似性,当它们被比较,以验证以前的实验结果,并了解如何识别是否上下文变量影响结果。方法:我们运行模拟,以了解在验证以前实验的结果时,比较复制结果的不同方法的表现。我们说明如何处理上下文引起的变化。为了做到这一点,我们分析了三组复制从我们自己的研究测试驱动的开发和测试技术。结果如下:p值和效应量的直接比较似乎不适合验证先前实验的结果和检查可能影响软件工程结果的变量。分析方法,如荟萃分析,应用于评估软件工程复制结果的相似性,并确定结果的差异。结论:在基线实验中获得的结果不应再被视为需要复制的结果,而应被视为一个更大的画面中的一小块证据,只有在组装许多小块以完成拼图之后才能出现。

英文摘要:

Context: It has been argued that software engineering replications are useful for verifying the results of previous experiments. However, it has not yet been agreed how to check whether the results hold across replications. Besides, some authors suggest that replications that do not verify the results of previous experiments can be used to identify contextual variables causing the discrepancies. Objective: Study how to assess the (dis)similarity of the results of SE replications when they are compared to verify the results of previous experiments and understand how to identify whether contextual variables are influencing results. Method: We run simulations to learn how different ways of comparing replication results behave when verifying the results of previous experiments. We illustrate how to deal with context-induced changes. To do this, we analyze three groups of replications from our own research on test-driven development and testing techniques. Results: The direct comparison of p-values and effect sizes does not appear to be suitable for verifying the results of previous experiments and examining the variables possibly affecting the results in software engineering. Analytical methods such as meta-analysis should be used to assess the similarity of software engineering replication results and identify discrepancies in results. Conclusion: The results achieved in baseline experiments should no longer be regarded as a result that needs to be reproduced, but as a small piece of evidence within a larger picture that only emerges after assembling many small pieces to complete the puzzle.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑