The general purpose of a scientific publication is the exchange and spread of knowledge. A publication usually reports a scientific result and tries to convince the reader that it is valid. With an ever-growing number of papers relying on computational methods that make use of large quantities of data and sophisticated statistical modeling techniques, a textual description of the result is often not enough for a publication to be transparent and reproducible. While there are efforts to encourage sharing of code and data, we currently lack conventions for linking data sources to a computational result that is stated in the main publication text or used to generate a figure or table. Thus, here I propose a data citation format that allows for an automatic reproduction of all computations. A data citation consists of a descriptor that refers to the functional program code and the input that generated the result. The input itself may be a set of other data citations, such that all data transformations, from the original data sources to the final result, are transparently expressed by a directed graph. Functions can be implemented in a variety of programming languages since data sources are expected to be stored in open and standardized text-based file formats. A publication is then an online file repository consisting of a Hypertext Markup Language (HTML) document and additional data and code source files, together with a summarization of all data sources, similar to a list of references in a bibliography.
Booz Allen Hamilton（博思艾伦咨询公司）；University of Maryland, Baltimore County（马里兰大学巴尔的摩县分校）
The field of bibliometrics, studying citations and behavior, is critical to the discussion of reproducibility. Citations are one of the primary incentive and reward systems for academic work, and so we desire to know if this incentive rewards reproducible work. Yet to the best of our knowledge, only one work has attempted to look at this combined space, concluding that non-reproducible work is more highly cited. We show that answering this question is more challenging than first proposed, and subtle issues can inhibit a robust conclusion. To make inferences with more robust behavior, we propos…
This work aims to identify classes of DOI mistakes by analysing the open bibliographic metadata available in Crossref, highlighting which publishers were responsible for such mistakes and how many of these incorrect DOIs could be corrected through automatic processes. By using a list of invalid cited DOIs gathered by OpenCitations while processing the OpenCitations Index of Crossref open DOI-to-DOI citations (COCI) in the past two years, we retrieved the citations in the January 2021 Crossref dump to such invalid DOIs. We processed these citations by keeping track of their validity and the pub…
Graduate School of Education, Stanford University（斯坦福大学教育研究生院）；Public Knowledge Project, Simon Fraser University（公共知识项目，西蒙弗雷泽大学）；DataCite, Hannover, Germany（DataCite，德国汉诺威）；California Digital Library, University of California Office of the President（加州数字图书馆，加州大学校长办公室）
Efforts to make research results open and reproducible are increasingly reflected by journal policies encouraging or mandating authors to provide data availability statements. As a consequence of this, there has been a strong uptake of data availability statements in recent literature. Nevertheless, it is still unclear what proportion of these statements actually contain well-formed links to data, for example via a URL or permanent identifier, and if there is an added value in providing such links. We consider 531,889 journal articles published by PLOS and BMC, develop an automatic system for …
The purpose of this paper is to update the review of Bornmann and Daniel (2008) presenting a narrative review of studies on citations in scientific documents. The current review covers 41 studies published between 2006 and 2018. Bornmann and Daniel (2008) focused on earlier years. The current review describes the (new) studies on citation content and context analyses as well as the studies that explore the citation motivation of scholars through surveys or interviews. One focus in this paper is on the technical developments in the last decade, such as the richer meta-data available and machine…
The trend toward open science increases the pressure on authors to provide access to the source code and data they used to compute the results reported in their scientific papers. Since sharing materials reproducibly is challenging, several projects have developed solutions to support the release of executable analyses alongside articles. We reviewed 11 applications that can assist researchers in adhering to reproducibility principles. The applications were found through a literature search and interactions with the reproducible research community. An application was included in our analysis i…
Citations are the cornerstone of knowledge propagation and the primary means of assessing the quality of research, as well as directing investments in science. Science is increasingly becoming "data-intensive", where large volumes of data are collected and analyzed to discover complex patterns through simulations and experiments, and most scientific reference works have been replaced by online curated datasets. Yet, given a dataset, there is no quantitative, consistent and established way of knowing how it has been used over time, who contributed to its curation, what results have been yielded…
Scientific research frequently involves the use of computational tools and methods. Providing thorough documentation, open-source code, and data -- the creation of reproducible computational research -- helps others understand a researcher's work. Here, we explore computational reproducibility, broadly, and from within the field of prognostics and health management (PHM). The adoption of reproducible computational research practices remains low across scientific disciplines and within PHM. Our text mining of more than 300 articles, from publications engaged in PHM research, showed that fewer t…
Open data and open-source software may be part of the solution to science's "reproducibility crisis", but they are insufficient to guarantee reproducibility. Requiring minimal end-user expertise, encapsulator creates a "time capsule" with reproducible code in a self-contained computational environment. encapsulator provides end-users with a fully-featured desktop environment for reproducible research.
Software citation has accelerated in astrophysics in the past decade, resulting in the field now having multiple trackable ways to cite computational methods. Yet most software authors do not specify how they would like their code to be cited, while others specify a citation method that is not easily tracked (or tracked at all) by most indexers. Two metadata file formats, codemeta.json and CITATION.cff, developed in 2016 and 2017 respectively, are useful for specifying how software should be cited. In 2020, the Astrophysics Source Code Library (ASCL, ascl.net) undertook a year-long effort to g…
Reproducible computational research (RCR) is the keystone of the scientific method for in silico analyses, packaging the transformation of raw data to published results. In addition to its role in research integrity, RCR has the capacity to significantly accelerate evaluation and reuse. This potential and wide-support for the FAIR principles have motivated interest in metadata standards supporting RCR. Metadata provides context and provenance to raw data and methods and is essential to both discovery and validation. Despite this shared connection with scientific data, few studies have explicit…
The rapid growth of scientific literature has made it difficult for the researchers to quickly learn about the developments in their respective fields. Scientific document summarization addresses this challenge by providing summaries of the important contributions of scientific papers. We present a framework for scientific summarization which takes advantage of the citations and the scientific discourse structure. Citation texts often lack the evidence and context to support the content of the cited paper and are even sometimes inaccurate. We first address the problem of inaccuracy of the cita…
University of Manchester（曼彻斯特大学）；Idiap Research Institute（伊迪亚普研究所）
Diagrams are often used in scholarly communication. We analyse a corpus of diagrams found in scholarly computational linguistics conference proceedings (ACL 2017), and find inclusion of a system diagram to be correlated with higher numbers of citations after 3 years. Inclusion of over three diagrams in this 8-page limit conference was found to correlate with a lower citation count. Focusing on neural network system diagrams, we find a correlation between highly cited papers and "good diagramming practice" quantified by level of compliance with a set of diagramming guidelines. Two diagram class…
Overcoming Barriers to Computational Reproducibility
Ludwig-Maximilians-Universität (LMU)（路德维希-马克西米利安大学）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；Weierstrass Institute for Applied Analysis and Stochastics（魏尔斯特拉斯应用分析与随机过程研究所）；Max Planck Institute for Demographic Research（马克斯·普朗克人口研究研究所）；Technical University of Munich（慕尼黑工业大学）；Medical University of Vienna（维也纳医科大学）；Lancaster University（兰卡斯特大学）；Clemson University（克莱姆森大学）；University of California, Berkeley（加州大学伯克利分校）；INRAE（法国国家农业、食品与环境研究院）；AgroParisTech（巴黎农业学院）；Université Paris-Saclay（巴黎萨克雷大学）
Recent reproducibility case studies have raised concerns showing that much of the deposited research has not been reproducible. One of their conclusions was that the way data repositories store research data and code cannot fully facilitate reproducibility due to the absence of a runtime environment needed for the code execution. New specialized reproducibility tools provide cloud-based computational environments for code encapsulation, thus enabling research portability and reproducibility. However, they do not often enable research discoverability, standardized data citation, or long-term ar…
University of California, San Diego（加州大学圣地亚哥分校）；University of Toronto（多伦多大学）；Dunlap Institute for Astronomy & Astrophysics, University of Toronto（多伦多大学邓普尔天文与天体物理研究所）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；FAIR Data Innovations Hub, California Medical Innovations Institute（公平数据创新中心，加州医疗创新研究所）；National Center for Supercomputing Applications, University of Illinois Urbana-Champaign（美国国家超级计算应用中心，伊利诺伊大学厄巴纳-香槟分校）；SciOS；University of California Berkeley（加州大学伯克利分校）；Center for Astrophysics | Harvard & Smithsonian（哈佛&史密森尼天体物理中心）；Astrophysics Source Code Library | University of Maryland College Park（天体物理学源代码库，马里兰大学帕克分校）；Universität Heidelberg, Zentrum für Astronomie (ZAH), Institut für Theoretische Astrophysik（海德堡大学，天文学中心（ZAH），理论天体物理研究所）；Institute for Cyber-Enabled Research, Michigan State University（密歇根州立大学网络使能研究所）；The US Research Software Engineer Association（美国研究软件工程师协会）；San Diego Supercomputer Center, University of California, San Diego（加州大学圣地亚哥分校圣地亚哥超级计算中心）；Center for Computational Astrophysics, Flatiron Institute（平顿研究所计算天体物理中心）
The Reproducible Research Platform establishes a unified open science environment bridging data and software lifecycles across disciplines, from proposal to publication
Scholars frequently employ relatedness measures to estimate the similarity between two different items (e.g., documents, authors, and institutes). Such relatedness measures are commonly based on overlapping references ($\textit{i.e.}$, bibliographic coupling) or citations ($\textit{i.e.}$, co-citation) and can then be used with cluster analysis to find boundaries between research fields. Unfortunately, calculating a relatedness measure is challenging, especially for a large number of items, because the computational complexity is greater than linear. We propose an alternative method for identi…
Reproducibility has been consistently identified as an important component of scientific research. Although there is a general consensus on the importance of reproducibility along with the other commonly used 'R' terminology (i.e., Replicability, Repeatability etc.), there is some disagreement on the usage of these terms, including conflicting definitions used by different parts of the research community. In this encyclopedia article, we explore the different definitions used in scientific literature (specifically pertaining to computational research), whether there is a need for a single stan…
