探索经验软件工程研究中的分叉路径花园：一个多宇宙分析
Exploring the Garden of Forking Paths in Empirical Software Engineering Research: A Multiverse Analysis
AI总结：
本文通过多宇宙分析揭示经验软件工程研究中方法学选择对结果的深远影响，提出结构化分类模型以提高分析的稳健性和可重复性。
摘要：
在经验软件工程（SE）研究中，研究者在数据处理方式、操作化选择以及统计模型拟合上具有相当大的自由度。Gelman和Loken将这种自由称为导致"分叉路径花园"。尽管这种自由常被视为优势，但它也对稳健性和可重复性构成威胁：即使分析决策的变异是合理的，也可能导致结论分歧。为了更好地理解这种风险，我们对一篇已发表的经验SE论文进行了所谓的多宇宙分析。我们选择的论文是一篇关于挖掘软件仓库的研究，因为MSR研究通常使用非平凡的统计模型来分析事后观察数据。在研究中，我们识别出九个关键的分析决策——每个都有至少一个同样有辩护力的替代方案，并系统地重新运行了所有3,072种由此产生的分析流水线，对原始数据集进行重新分析。有趣的是，只有6个宇宙（<0.2%）复现了已发表的结果；大多数产生了定性不同的，甚至有时相反的发现。这项对常用于经验软件工程数据的分析方法的案例研究揭示了方法学选择对结果的影响可能比人们通常意识到的要深远。因此，我们建议SE研究者在标准报告中补充跨合理分析变体的稳健性检查，或者至少明确justify每个分析决策。我们提出了一种结构化的分类模型，以帮助分类和改进方法学选择的辩护。其次，我们展示了多宇宙分析是SE研究者方法论工具箱中的实用工具，有助于产生更可靠、可重复的科学。
英文摘要：
In empirical software engineering (SE) research, researchers have considerable freedom to decide how to process data, what operationalizations to use, and which statistical model to fit. Gelman and Loken refer to this freedom as leading to a "garden of forking paths". Although this freedom is often seen as an advantage, it also poses a threat to robustness and replicability: variations in analytical decisions, even when justifiable, can lead to divergent conclusions. To better understand this risk, we conducted a so-called multiverse analysis on a published empirical SE paper. The paper we picked is a Mining Software Repositories study, as MSR studies commonly use non-trivial statistical models to analyze post-hoc, observational data. In the study, we identified nine pivotal analytical decisions-each with at least one equally defensible alternative and systematically reran all the 3,072 resulting analysis pipelines on the original dataset. Interestingly, only 6 of these universes (<0.2%) reproduced the published results; the overwhelming majority produced qualitatively different, and sometimes even opposite, findings. This case study of a data analytical method commonly applied to empirical software engineering data reveals how methodological choices can exert a more profound influence on outcomes than is often acknowledged. We therefore advocate that SE researchers complement standard reporting with robustness checks across plausible analysis variants or, at least, explicitly justify each analytical decision. We propose a structured classification model to help classify and improve justification for methodological choices. Secondly, we show how the multiverse analysis is a practical tool in the methodological arsenal of SE researchers, one that can help produce more reliable, reproducible science.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
