arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2103.12793cs.SEcs.DL
收藏

关于研究代码质量和执行的大规模研究

A large-scale study on research code quality and execution

Ana Trisovic, Matthew K. Lau, Thomas Pasquier, Mercè Crosas

摘要:

本文介绍了一项研究的质量和执行的研究代码从公开的复制数据集在哈佛Dataverse存储库。研究代码通常由一组科学家创建,并与学术论文一起发布,以促进研究的透明度和可重复性。在这项研究中,我们定义了十个问题来解决影响研究再现性和重复使用的方面。首先,我们检索和分析了2010年至2020年发布的2000多个复制数据集,其中包含9000多个唯一的R文件。其次,我们在一个干净的运行时环境中执行代码,以评估其重用的容易性。识别了常见的编码错误,其中一些通过自动代码清理来帮助代码执行。我们发现,74%的R文件在初始执行时崩溃,而56%在应用代码清理时崩溃,这表明通过良好的编码实践可以防止许多错误。我们还分析了日志集合中的复制数据集,并讨论了日志策略严格性对代码重执行率的影响。最后,根据我们的研究结果,我们提出了一套针对研究人员,期刊和知识库的代码传播的建议。

英文摘要:

This article presents a study on the quality and execution of research code from publicly-available replication datasets at the Harvard Dataverse repository. Research code is typically created by a group of scientists and published together with academic papers to facilitate research transparency and reproducibility. For this study, we define ten questions to address aspects impacting research reproducibility and reuse. First, we retrieve and analyze more than 2000 replication datasets with over 9000 unique R files published from 2010 to 2020. Second, we execute the code in a clean runtime environment to assess its ease of reuse. Common coding errors were identified, and some of them were solved with automatic code cleaning to aid code execution. We find that 74\% of R files crashed in the initial execution, while 56\% crashed when code cleaning was applied, showing that many errors can be prevented with good coding practices. We also analyze the replication datasets from journals' collections and discuss the impact of the journal policy strictness on the code re-execution rate. Finally, based on our results, we propose a set of recommendations for code dissemination aimed at researchers, journals, and repositories.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑