软件工程中的系统文献综述--利用Cohen's Kappa统计量改进研究选择过程
Systematic Literature Reviews in Software Engineering -- Enhancement of the Study Selection Process using Cohen's Kappa Statistic
摘要：
背景：系统性文献综述（SLR）依赖于严格和可审计的方法，以最大限度地减少偏倚并确保可靠性。当使用一组纳入/排除标准选择研究时，会出现一种常见的偏倚。这种偏倚可以通过双重修订来减少，这使得选择过程更加耗时，并且仍然容易产生偏倚，这取决于每个研究人员如何解释入选/排除标准。目的：为了减少研究选择过程中的偏差和时间花费，本文提出了一种基于Cohen's Kappa统计量的研究选择过程。我们已经定义了一个迭代过程的基础上使用这个统计量，在此期间的标准进行细化，直到获得几乎完美的协议（k>0.8）。在这一点上，两位研究人员以相同的方式解释选择标准，因此，偏差减少了。从这一协议开始，可以消除双重审查;因此，所花费的时间大大缩短。方法：通过对2005年至2018年出版的软件工程作品进行三级研究，证明了这种迭代过程选择研究的可行性。结果如下：在研究选择过程中节省的时间为28%（152项研究），如果研究数量足够大，节省的时间趋于渐近50%。结论：研究人员和学生在进行SLR时可以利用这种迭代过程来选择研究，以减少对纳入和排除标准解释的偏倚。它对于资源很少的研究特别有用。
英文摘要：
Context: Systematic literature reviews (SLRs) rely on a rigorous and auditable methodology for minimizing biases and ensuring reliability. A common kind of bias arises when selecting studies using a set of inclusion/exclusion criteria. This bias can be decreased through dual revision, which makes the selection process more time-consuming and remains prone to generating bias depending on how each researcher interprets the inclusion/exclusion criteria. Objective: To reduce the bias and time spent in the study selection process, this paper presents a process for selecting studies based on the use of Cohen's Kappa statistic. We have defined an iterative process based on the use of this statistic during which the criteria are refined until obtain almost perfect agreement (k>0.8). At this point, the two researchers interpret the selection criteria in the same way, and thus, the bias is reduced. Starting from this agreement, dual review can be eliminated; consequently, the time spent is drastically shortened. Method: The feasibility of this iterative process for selecting studies is demonstrated through a tertiary study in the area of software engineering on works that were published from 2005 to 2018. Results: The time saved in the study selection process was 28% (for 152 studies) and if the number of studies is sufficiently large, the time saved tend asymptotically to 50%. Conclusions: Researchers and students may take advantage of this iterative process for selecting studies when conducting SLRs to reduce bias in the interpretation of inclusion and exclusion criteria. It is especially useful for research with few resources.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
