统计方法研究中模拟研究的可复现性:RepliSims 项目
Replicability of Simulation Studies for the Investigation of Statistical Methods: The RepliSims Project
AI总结:
RepliSims 项目通过复现八项高被引统计模拟研究,发现可复现性差异显著,主要受信息缺失和来源不可持续阻碍,并建议提供开放代码、数据及透明报告以提升可复现性。
摘要:
评估统计方法性能的模拟研究结果通常被视为具有可操作性,因此可能对实证研究的实施方式产生重大影响。然而,迄今为止,关于统计模拟研究的可重复性和可复现性的证据仍然有限。为此,本研究选取了八项高被引的统计模拟研究,并由接受过定量方法学正式培训的复现团队对其可复现性进行评估。这些团队从原始论文中寻找相关信息,并据此编写模拟代码,旨在复现原始结果。主要结局指标是基于原始论文所报告信息的可复现可行性。可复现性差异很大:一些原始研究提供了详细信息,使得结果几乎可以被完美复现;而另一些研究则未能提供足够的信息来实施任何所报告的模拟。复现者不得不就原始研究中缺失或含糊不清的信息、错误处理方式以及软件环境做出选择。促进复现的因素包括代码的公开可用性,以及通过图形、公式、结构化文本和可公开访问的附加资源(如技术报告)对数据生成过程和方法进行描述。统计模拟研究的可复现性主要受到信息缺失和信息来源可持续性不足的阻碍。通过提供开放代码和数据作为论文的补充材料,模拟研究可以实现可重复性。此外,模拟研究应当进行透明报告,将所有相关信息呈现在研究论文本身或易于获取的补充材料中,以便实现可复现性。
英文摘要:
Results of simulation studies evaluating the performance of statistical methods are often considered actionable and thus can have a major impact on the way empirical research is implemented. However, so far there is limited evidence about the reproducibility and replicability of statistical simulation studies. Therefore, eight highly cited statistical simulation studies were selected, and their replicability was assessed by teams of replicators with formal training in quantitative methodology. The teams found relevant information in the original publications and used it to write simulation code with the aim of replicating the results. The primary outcome was the feasibility of replicability based on reported information in the original publications. Replicability varied greatly: Some original studies provided detailed information leading to almost perfect replication of results, whereas other studies did not provide enough information to implement any of the reported simulations. Replicators had to make choices regarding missing or ambiguous information in the original studies, error handling, and software environment. Factors facilitating replication included public availability of code, and descriptions of the data-generating procedure and methods in graphs, formulas, structured text, and publicly accessible additional resources such as technical reports. Replicability of statistical simulation studies was mainly impeded by lack of information and sustainability of information sources. Reproducibility could be achieved for simulation studies by providing open code and data as a supplement to the publication. Additionally, simulation studies should be transparently reported with all relevant information either in the research paper itself or in easily accessible supplementary material to allow for replicability.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。