通过编码者间一致性提升软件工程定性研究的可靠性：使用Krippendorff's α与Atlas.ti的指南
Reliability in Software Engineering Qualitative Research through Inter-Coder Agreement: A guide using Krippendorff's $α$ & Atlas.ti
AI总结：
本文针对软件工程定性研究中编码可靠性不足的问题，提出统一的Krippendorff's α数学框架，并结合Atlas.ti给出在DevOps文化案例中的详细应用指南，以提升实证研究的质量与可信度。
摘要：
近年来，使用定性数据分析（例如案例研究、访谈调查和扎根理论研究）的实证软件工程研究日益增多。然而，尽管存在多种被称为编码者间一致性（ICA）的统计技术可用于分析团队编码中的共识，但这类研究大多未深入探讨研究结果的可靠性与效度，尤其是这些方法论所依赖的编码可靠性。本文旨在建立一个新颖的理论框架，为开展此类效度分析提供方法论路径。该框架基于一组用于衡量不同编码者在判断共同事项时所达成一致程度的系数。我们分析了不同的可靠性系数，并提供了详细的计算示例，特别关注Krippendorff's α系数。我们系统梳理了文献中报告的Krippendorff's α的若干变体，并提出了一个新颖的通用数学框架，通过一个统一的α系数将所有变体整合在一起。最后，本文提供了在一个关于DevOps文化的大型案例研究中应用该理论框架的详细指南。我们解释了如何使用Atlas.ti这类广泛使用的定性分析软件工具来计算和解释α系数。我们期望这项工作能够帮助实证研究者，尤其是软件工程领域的研究者，提高其研究的质量和可信度。
英文摘要：
In recent years, the research on empirical software engineering that uses qualitative data analysis (e.g., cases studies, interview surveys, and grounded theory studies) is increasing. However, most of this research does not deep into the reliability and validity of findings, specifically in the reliability of coding in which these methodologies rely on, despite there exist a variety of statistical techniques known as Inter-Coder Agreement (ICA) for analyzing consensus in team coding. This paper aims to establish a novel theoretical framework that enables a methodological approach for conducting this validity analysis. This framework is based on a set of coefficients for measuring the degree of agreement that different coders achieve when judging a common matter. We analyze different reliability coefficients and provide detailed examples of calculation, with special attention to Krippendorff's $α$ coefficients. We systematically review several variants of Krippendorff's $α$ reported in the literature and provide a novel common mathematical framework in which all of them are unified through a universal $α$ coefficient. Finally, this paper provides a detailed guide of the use of this theoretical framework in a large case study on DevOps culture. We explain how $α$ coefficients are computed and interpreted using a widely used software tool for qualitative analysis like Atlas.ti. We expect that this work will help empirical researchers, particularly in software engineering, to improve the quality and trustworthiness of their studies.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
