arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2408.07594cs.SE
收藏

软件工程实验中的交叉设计:分析现状综述

Crossover Designs in Software Engineering Experiments: Review of the State of Analysis

Julian Frattini, Davide Fucci, Sira Vegas

AI总结:

本文通过前向雪球抽样评估136篇SE文献中的67项交叉实验,发现分析效度虽有提升,但仅29.5%的效度威胁被恰当处理,残留效应建模尤为不足。

摘要:

实验是任何经验性学科中进行因果推断的重要方法。交叉设计实验在软件工程(SE)研究中十分常见。在这类实验中,受试者以不同顺序接受不止一种处理。该设计增加了所获得的数据量,并能处理受试者差异性,但也会带来对内部效度的威胁,例如学习效应和残留效应。Vegas 等人曾综述了 SE 研究中交叉设计的实践现状,并就如何在数据分析中应对其威胁、同时继续发挥其优势提供了指南。本文中,我们反思这些指南的影响,并综述 2015 年至 2024 年 3 月期间 SE 出版物中交叉设计实验的分析现状。为此,我们通过对该指南进行前向雪球抽样,调查了 136 篇出版物,这些文献报告了 67 项交叉设计实验,并依据所提供的指南评估其数据分析。结果表明,与最初的分析现状相比,数据分析的效度有所改善。然而,尽管已有明确指南,所有效度威胁中仅有 29.5% 得到恰当处理。成熟威胁和最优顺序威胁在我们样本的全部研究中分别有 35.8% 和 38.8% 得到恰当处理,而残留威胁仅在约 3% 的观察案例中被建模。对分析指南的依从性不足,危及从交叉设计实验中得出结论的效度。

英文摘要:

Experimentation is an essential method for causal inference in any empirical discipline. Crossover-design experiments are common in Software Engineering (SE) research. In these, subjects apply more than one treatment in different orders. This design increases the amount of obtained data and deals with subject variability but introduces threats to internal validity like the learning and carryover effect. Vegas et al. reviewed the state of practice for crossover designs in SE research and provided guidelines on how to address its threats during data analysis while still harnessing its benefits. In this paper, we reflect on the impact of these guidelines and review the state of analysis of crossover design experiments in SE publications between 2015 and March 2024. To this end, by conducting a forward snowballing of the guidelines, we survey 136 publications reporting 67 crossover-design experiments and evaluate their data analysis against the provided guidelines. The results show that the validity of data analyses has improved compared to the original state of analysis. Still, despite the explicit guidelines, only 29.5% of all threats to validity were addressed properly. While the maturation and the optimal sequence threats are properly addressed in 35.8% and 38.8% of all studies in our sample respectively, the carryover threat is only modeled in about 3% of the observed cases. The lack of adherence to the analysis guidelines threatens the validity of the conclusions drawn from crossover design experiments

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑