AI 中文总结
该研究针对软件工程复制研究评估标准不明确不一致的问题，通过系统综述找到10项研究，分析发现显著异质性，进而提出基于多方面考量的有原则框架，能减少模糊性、提高可比性，助力可靠证据积累。
AI 中文摘要
背景：软件工程中的复制研究日益普遍，但由于评估常依赖定义模糊或临时的标准，其解释仍不确定且不一致。目的：记录实证软件工程中复制研究结果目前如何评估，识别因标准不一致产生的问题，并提出有原则的有意义评估框架。方法：对复制研究进行系统综述，搜索涵盖2021年至2025年近期实证软件工程复制。分析评估标准的异质性、逻辑一致性及与既定统计原则的一致性。结果：共找到10项复制研究。分析显示评估实践存在显著异质性，类似数据采用矛盾标准，对测量不确定性认识有限且缺乏共享标准。我们提出基于统计、方法和测量考量的框架并举例说明。结论：采用一致且透明的评估原则可减少软件工程复制研究中的模糊性，提高可比性并支持更可靠的证据积累。
英文摘要
Background: Replication studies in software engineering are increasingly common, yet their interpretation remains uncertain and inconsistent because assessments frequently rely on loosely defined or ad hoc criteria. Aim: This study aims to document how replication study outcomes are currently assessed in empirical software engineering, identify problems arising from inconsistent criteria and propose a principled framework for meaningful evaluation. Method: We conducted a systematic review of replication studies, with the search covering recent empirical software engineering replications (2021--2025). For each study, we extracted the criteria used to assess replication outcomes and analysed these for heterogeneity, logical consistency, and alignment with established statistical principles. Results: A total of 10 replication studies were located. The analysis reveals substantial heterogeneity in assessment practices, with contradictory criteria applied to similar data, limited acknowledgement of measurement uncertainty, and an absence of shared standards. We propose a principled framework grounded in statistical, methodological, and measurement considerations, and demonstrate its application through worked examples. Conclusions: Adopting consistent and transparent assessment principles would reduce ambiguity, improve comparability, and support more reliable evidence accumulation in software engineering replication research.
CommentsIn International Conference on Evaluation and Assessment in Software Engineering (EASE 2026), June 09--12, 2026, Glasgow, United Kingdom. ACM, New York, NY, USA, 6 pages
