Reproducibility has become an intensely debated topic in NLP and ML over recent years, but no commonly accepted way of assessing reproducibility, let alone quantifying it, has so far emerged. The assumption has been that wider scientific reproducibility terminology and definitions are not applicable to NLP/ML, with the result that many different terms and definitions have been proposed, some diametrically opposed. In this paper, we test this assumption, by taking the standard terminology and definitions from metrology and applying them directly to NLP/ML. We find that we are able to straightforwardly derive a practical framework for assessing reproducibility which has the desirable property of yielding a quantified degree of reproducibility that is comparable across different reproduction studies.
Reproducibility in NLP: What Have We Learned from the Checklist?
Allen Institute for Artificial Intelligence（艾伦人工智能研究所）；Paul G. Allen School of Computer Science & Engineering, University of Washington（华盛顿大学保罗·G·艾伦计算机科学与工程学院）
Lessons in Reproducibility: Insights from NLP Studies in Materials Science
Cohere（Cohere公司）；University of Toronto（多伦多大学）；Toyota Research Institute（丰田研究院）；Lawrence Berkeley National Laboratory（劳伦斯伯克利国家实验室）；University of Washington（华盛顿大学）
Missing Information, Unresponsive Authors, Experimental Flaws: The Impossibility of Assessing the Reproducibility of Previous Human Evaluations in NLP
ADAPT/DCU（ADAPT中心/都柏林城市大学）；University of Aberdeen（阿伯丁大学）；Bielefeld University（比勒费尔德大学）；Dataminr（Dataminr公司）；Edinburgh Napier University（爱丁堡龙比亚大学）；Georgia Tech（佐治亚理工学院）；Google Research（谷歌研究院）；Groningen University（格罗宁根大学）；Heriot-Watt University（赫瑞-瓦特大学）；Charles University Prague（布拉格查理大学）；Tohoku University（东北大学）；Peking University（北京大学）；Technological University Dublin（都柏林理工大学）；Tilburg University（蒂尔堡大学）；trivago（trivago公司）；University of Applied Sciences Darmstadt（达姆施塔特应用科学大学）；University of Illinois Chicago（伊利诺伊大学芝加哥分校）；Universidade de Santiago de Compostela（圣地亚哥德孔波斯特拉大学）；Universitat Pompeu Fabra（庞培法布拉大学）；Utrecht University（乌得勒支大学）；Zurich University of Applied Sciences（苏黎世应用科学大学）
Systematic Exploration of Dialogue Summarization Approaches for Reproducibility, Comparative Assessment, and Methodological Innovations for Advancing Natural Language Processing in Abstractive Summarization
SRM Institute of Science and Technology（SRM科学技术学院）；SRM University, AP（SRM大学（安得拉邦））
