机器学习评估的可靠性——即重复模型训练运行中观测到的评估分数的一致性——受到多种可被视为测量噪声的非确定性来源的影响。当前为了强制实现研究结果的可复现性而去除噪声的趋势，忽视了实现层面的固有非确定性，也忽略了算法噪声因子与数据属性之间关键的交互效应，这限制了可从此类实验中得出的结论的范围。我们并未去除噪声，而是提出将多种方差来源（包括它们与数据属性的交互作用）纳入机器学习评估的显著性与可靠性分析中，旨在得出超越已训练模型特定实例的推论。我们展示了如何使用线性混合效应模型（LMEMs，linear mixed effects models）分析性能评估分数，并通过广义似然比检验（GLRT，generalized likelihood ratio test）开展统计推断。这使我们能够将元参数变化等任意噪声来源纳入统计显著性检验，并评估以数据属性为条件的性能差异。此外，方差成分分析（VCA，variance component analysis）可用于分析各噪声来源对总方差的贡献，并通过实质方差与总方差的比值计算可靠性系数。
英文摘要：
Reliability of machine learning evaluation -- the consistency of observed evaluation scores across replicated model training runs -- is affected by several sources of nondeterminism which can be regarded as measurement noise. Current tendencies to remove noise in order to enforce reproducibility of research results neglect inherent nondeterminism at the implementation level and disregard crucial interaction effects between algorithmic noise factors and data properties. This limits the scope of conclusions that can be drawn from such experiments. Instead of removing noise, we propose to incorporate several sources of variance, including their interaction with data properties, into an analysis of significance and reliability of machine learning evaluation, with the aim to draw inferences beyond particular instances of trained models. We show how to use linear mixed effects models (LMEMs) to analyze performance evaluation scores, and to conduct statistical inference with a generalized likelihood ratio test (GLRT). This allows us to incorporate arbitrary sources of noise like meta-parameter variations into statistical significance testing, and to assess performance differences conditional on data properties. Furthermore, a variance component analysis (VCA) enables the analysis of the contribution of noise sources to overall variance and the computation of a reliability coefficient by the ratio of substantial to total variance.
Shanghai Jiao Tong University（上海交通大学）；New York University（纽约大学）
The growing size of modern data brings many new challenges to existing statistical inference methodologies and theories, and calls for the development of distributed inferential approaches. This paper studies distributed inference for linear support vector machine (SVM) for the binary classification task. Despite a vast literature on SVM, much less is known about the inferential properties of SVM, especially in a distributed setting. In this paper, we propose a multi-round distributed linear-type (MDL) estimator for conducting inference for linear SVM. The proposed estimator is computationally…
Prediction, where observed data is used to quantify uncertainty about a future observation, is a fundamental problem in statistics. Prediction sets with coverage probability guarantees are a common solution, but these do not provide probabilistic uncertainty quantification in the sense of assigning beliefs to relevant assertions about the future observable. Alternatively, we recommend the use of a {\em probabilistic predictor}, a data-dependent (imprecise) probability distribution for the to-be-predicted observation given the observed data. It is essential that the probabilistic predictor be r…
The soft-margin support vector machine (SVM) is a ubiquitous tool for prediction of binary-response data. However, the SVM is characterized entirely via a numerical optimization problem, rather than a probability model, and thus does not directly generate probabilistic inferential statements as outputs. We consider a probabilistic regression model for binary-response data that is based on the optimization problem that characterizes the SVM. Under weak regularity assumptions, we prove that the maximum likelihood estimate (MLE) of our model exists, and that it is consistent and asymptotically no…
In the early 2010s, a crisis of reproducibility rocked the field of psychology. Following a period of reflection, the field has responded with radical reform of its scientific practices. More recently, similar questions about the reproducibility of machine learning research have also come to the fore. In this short paper, we present select ideas from psychology's reformation, translating them into relevance for a machine learning audience.
Heinz-Nixdorf Chair for Distributed Information Systems（海因茨-尼克斯多夫分布式信息系统讲席）；Michael Stifel Center Jena（耶拿米夏埃尔·施蒂费尔中心）；Friedrich Schiller University Jena（耶拿弗里德里希·席勒大学）
Machine learning (ML) is an increasingly important scientific tool supporting decision making and knowledge generation in numerous fields. With this, it also becomes more and more important that the results of ML experiments are reproducible. Unfortunately, that often is not the case. Rather, ML, similar to many other disciplines, faces a reproducibility crisis. In this paper, we describe our goals and initial steps in supporting the end-to-end reproducibility of ML pipelines. We investigate which factors beyond the availability of source code and datasets influence reproducibility of ML exper…
McGill University（麦吉尔大学）；Facebook AI Research（Facebook人工智能研究院）；CIFAR（加拿大高等研究院）；Université de Montréal（蒙特利尔大学）；Yahoo! Research（雅虎研究院）；Télécom Paris（巴黎电信学院）；Institut Polytechnique de France（法国理工学院）；University of Washington（华盛顿大学）；Apple（苹果公司）；Google（谷歌）
One of the challenges in machine learning research is to ensure that presented and published results are sound and reliable. Reproducibility, that is obtaining similar results as presented in a paper or talk, using the same code and data (when available), is a necessary step to verify the reliability of research findings. Reproducibility is also an important step to promote open and accessible research, thereby allowing the scientific community to quickly integrate new findings and convert ideas to practice. Reproducibility also promotes the use of robust experimental workflows, which potentia…
Exploring Non-Linear Effects of Built Environment on Travel Using an Integrated Machine Learning and Inferential Modeling Approach: A Three-Wave Repeated Cross-Sectional Study
Massachusetts Institute of Technology（麻省理工学院）；University of Texas at Austin（得克萨斯大学奥斯汀分校）
Hypothesis Testing and Machine Learning: Interpreting Variable Effects in Deep Artificial Neural Networks using Cohen's f2
University of South Carolina（南卡罗来纳大学）；University of Kassel（卡塞尔大学）；University of Wales（威尔士大学）；Technical University Munich（慕尼黑工业大学）；University of Newcastle upon Tyne（纽卡斯尔大学）；Università per Stranieri di Perugia（佩鲁贾外国人大学）
Sorbonne Université（索邦大学）；Institut du Cerveau - Paris Brain Institute - ICM（巴黎脑科学院）；CNRS（法国国家科学研究中心）；Inria（法国国家信息与自动化研究所）；Inserm（法国国家健康与医学研究院）；AP-HP（巴黎公共医疗救助机构）；Hôpital de la Pitié-Salpêtrière（巴黎硝石库慈善医院）；University of Twente（特文特大学）
Reproducibility is a cornerstone of science, as the replication of findings is the process through which they become knowledge. It is widely considered that many fields of science are undergoing a reproducibility crisis. This has led to the publications of various guidelines in order to improve research reproducibility. This didactic chapter intends at being an introduction to reproducibility for researchers in the field of machine learning for medical imaging. We first distinguish between different types of reproducibility. For each of them, we aim at defining it, at describing the requiremen…
Statistical machine learning models trained with stochastic gradient algorithms are increasingly being deployed in critical scientific applications. However, computing the stochastic gradient in several such applications is highly expensive or even impossible at times. In such cases, derivative-free or zeroth-order algorithms are used. An important question which has thus far not been addressed sufficiently in the statistical machine learning literature is that of equipping stochastic zeroth-order algorithms with practical yet rigorous inferential capabilities so that we not only have point es…
Massachusetts Institute of Technology（麻省理工学院）；University of Toronto（多伦多大学）；Evidation Health, Inc.（Evidation健康公司）；New York University（纽约大学）；Vector Institute（向量研究所）
Machine learning algorithms designed to characterize, monitor, and intervene on human health (ML4H) are expected to perform safely and reliably when operating at scale, potentially outside strict human supervision. This requirement warrants a stricter attention to issues of reproducibility than other fields of machine learning. In this work, we conduct a systematic evaluation of over 100 recently published ML4H research papers along several dimensions related to reproducibility. We find that the field of ML4H compares poorly to more established machine learning fields, particularly concerning …
Computational reproducibility is a growing problem that has been extensively studied among computational researchers and within the signal processing and machine learning research community. However, with the changing landscape of signal processing and machine learning research come new obstacles and unseen challenges in creating reproducible experiments. Due to these new challenges most computational experiments have become difficult, if not impossible, to be reproduced by an independent researcher. In 2016 a survey conducted by the journal Nature found that 50% of researchers were unable to …
Booz Allen Hamilton（博兹·艾伦·汉密尔顿公司）；University of Maryland, Baltimore County（马里兰大学巴尔的摩分校）
There has been increasing concern within the machine learning community that we are in a reproducibility crisis. As many have begun to work on this problem, all work we are aware of treat the issue of reproducibility as an intrinsic binary property: a paper is or is not reproducible. Instead, we consider modeling the reproducibility of a paper as a survival analysis problem. We argue that this perspective represents a more accurate model of the underlying meta-science question of reproducible research, and we show how a survival analysis allows us to draw new insights that better explain prior…
Recent years have seen a substantial development of quantitative methods, mostly led by the computer science community with the goal of developing better machine learning applications, mainly focused on predictive modeling. However, economic, management, and technology forecasting research has so far been hesitant to apply predictive modeling techniques and workflows. In this paper, we introduce a machine learning (ML) approach to quantitative analysis geared towards optimizing the predictive performance, contrasting it with standard practices inferential statistics, which focus on producing g…
Reproducibility, energy efficiency and performance of pseudorandom number generators in machine learning: a comparative study of python, numpy, tensorflow, and pytorch implementations
Reproducibility of modeling is a problem that exists for any machine learning practitioner, whether in industry or academia. The consequences of an irreproducible model can include significant financial costs, lost time, and even loss of personal reputation (if results prove unable to be replicated). This paper will first discuss the problems we have encountered while building a variety of machine learning models, and subsequently describe the framework we built to tackle the problem of model reproducibility. The framework is comprised of four main components (data, feature, scoring, and evalu…
Purdue University（普渡大学）；Loyola University Chicago（芝加哥洛约拉大学）
Machine learning techniques are becoming a fundamental tool for scientific and engineering progress. These techniques are applied in contexts as diverse as astronomy and spam filtering. However, correctly applying these techniques requires careful engineering. Much attention has been paid to the technical potential; relatively little attention has been paid to the software engineering process required to bring research-based machine learning techniques into practical utility. Technology companies have supported the engineering community through machine learning frameworks such as TensorFLow an…
Old Dominion University（欧道明大学）；Pennsylvania State University（宾夕法尼亚州立大学）；Texas A&M University（德克萨斯A&M大学）
In recent years, significant effort has been invested verifying the reproducibility and robustness of research claims in social and behavioral sciences (SBS), much of which has involved resource-intensive replication projects. In this paper, we investigate prediction of the reproducibility of SBS papers using machine learning methods based on a set of features. We propose a framework that extracts five types of features from scholarly work that can be used to support assessments of reproducibility of published research claims. Bibliometric features, venue features, and author features are coll…
University of Bonn（波恩大学）；Fraunhofer Institute for Intelligent Analysis and Information Systems (IAIS)（弗劳恩霍夫智能分析与信息系统研究所（IAIS））；Fraunhofer Institute for Algorithms and Scientific Computing (SCAI)（弗劳恩霍夫算法与科学计算研究所（SCAI））
There is an emerging trend of embedding knowledge graphs (KGs) in continuous vector spaces in order to use those for machine learning tasks. Recently, many knowledge graph embedding (KGE) models have been proposed that learn low dimensional representations while trying to maintain the structural properties of the KGs such as the similarity of nodes depending on their edges to other nodes. KGEs can be used to address tasks within KGs such as the prediction of novel links and the disambiguation of entities. They can also be used for downstream tasks like question answering and fact-checking. Ove…
Booz Allen Hamilton（博思艾伦汉密尔顿）；University of Maryland, Baltimore County（马里兰大学巴尔的摩县分校）
What makes a paper independently reproducible? Debates on reproducibility center around intuition or assumptions but lack empirical results. Our field focuses on releasing code, which is important, but is not sufficient for determining reproducibility. We take the first step toward a quantifiable answer by manually attempting to implement 255 papers published from 1984 until 2017, recording features of each paper, and performing statistical analysis of the results. For each paper, we did not look at the authors code, if released, in order to prevent bias toward discrepancies between code and p…
Technical University of Darmstadt（达姆施塔特工业大学）；Bielefeld University（比勒费尔德大学）
Reproducibility is of utmost concern in machine learning and natural language processing (NLP). In the field of natural language generation (especially machine translation), the seminal paper of Post (2018) has pointed out problems of reproducibility of the dominant metric, BLEU, at the time of publication. Nowadays, BERT-based evaluation metrics considerably outperform BLEU. In this paper, we ask whether results and claims from four recent BERT-based metrics can be reproduced. We find that reproduction of claims and results often fails because of (i) heavy undocumented preprocessing involved …
Booz Allen Hamilton（博思艾伦咨询公司）；University of Maryland, Baltimore County（马里兰大学巴尔的摩县分校）
As reproducibility becomes a greater concern, conferences have largely converged to a strategy of asking reviewers to indicate whether code was attached to a submission. This is part of a larger trend of taking action based on assumed ideals, without studying if those actions will yield the desired outcome. Our argument is that this focus on code for replication is misguided if we want to improve the state of reproducible research. This focus can be harmful -- we should not force code to be submitted. There is a lack of evidence for effective actions taken by conferences to encourage and rewar…
Institute of Mathematics and Computer Science（数学与计算机科学研究所）；University of São Paulo（圣保罗大学）；São Carlos School of Engineering（圣卡洛斯工程学院）；Institute of Informatics（信息学研究所）；Federal University of Goiás（戈亚斯联邦大学）
Extracting knowledge from unlabeled texts using machine learning algorithms can be complex. Document categorization and information retrieval are two applications that may benefit from unsupervised learning (e.g., text clustering and topic modeling), including exploratory data analysis. However, the unsupervised learning paradigm poses reproducibility issues. The initialization can lead to variability depending on the machine learning algorithm. Furthermore, the distortions can be misleading when regarding cluster geometry. Amongst the causes, the presence of outliers and anomalies can be a de…
Concerns about the reproducibility of deep learning research are more prominent than ever, with no clear solution in sight. The relevance of machine learning research can only be improved if we also employ empirical rigor that incorporates reproducibility guidelines, especially so in the medical imaging field. The Medical Imaging with Deep Learning (MIDL) conference has made advancements in this direction by advocating open access, and recently also recommending authors to make their code public - both aspects being adopted by the majority of the conference submissions. This helps the reproduc…
Reproducibility Assessment of Magnetic Resonance Spectroscopy of Pregenual Anterior Cingulate Cortex across Sessions and Vendors via the Cloud Computing Platform CloudBrain-MRS
National Institute for Data Science in Health and Medicine, Xiamen University（厦门大学健康与医学数据科学国家研究院）；College of Ocean & Earth Sciences, Xiamen University（厦门大学海洋与地球学院）；Fujian Provincial Key Laboratory of Plasma and Magnetic Resonance, Xiamen University（厦门大学福建省等离子体与磁共振重点实验室）；Department of Electronic Science, Xiamen University（厦门大学电子科学系）；Clinical and Technical Support, Philips Healthcare（飞利浦医疗保健临床与技术支持部）；School of Economics and Management, Xiamen University of Technology（厦门理工学院经济与管理学院）；Department of Radiology, the First Hospital Affiliated to Xiamen University（厦门大学附属第一医院放射科）；China Mobile (Suzhou) Software Technology Company Limited（中国移动苏州软件技术有限公司）；School of Computer and Information Engineering, Xiamen University of Technology（厦门理工学院计算机与信息工程学院）
Sources of Irreproducibility in Machine Learning: A Review
Norwegian University of Science and Technology（挪威科技大学）；Aneo AS（Aneo公司）；San Diego Supercomputer Center, UC San Diego（加州大学圣地亚哥分校圣地亚哥超级计算中心）；Information Sciences Institute, USC（南加州大学信息科学研究所）
The University of Michigan（密歇根大学）；The University of Pennsylvania（宾夕法尼亚大学）
Big data repositories from online learning platforms such as Massive Open Online Courses (MOOCs) represent an unprecedented opportunity to advance research on education at scale and impact a global population of learners. To date, such research has been hindered by poor reproducibility and a lack of replication, largely due to three types of barriers: experimental, inferential, and data. We present a novel system for large-scale computational research, the MOOC Replication Framework (MORF), to jointly address these barriers. We discuss MORF's architecture, an open-source platform-as-a-service …
Eawag（瑞士联邦水科学与技术研究所）；ETHZ（苏黎世联邦理工学院）；Sapienza University of Rome（罗马第一大学）；Swiss Federal Institute for Forest, Snow, and Landscape Research WSL（瑞士联邦森林、雪与景观研究所）；EPFL（洛桑联邦理工学院）
We applied machine learning methods to predict chemical hazards focusing on fish acute toxicity across taxa. We analyzed the relevance of taxonomy and experimental setup, showing that taking them into account can lead to considerable improvements in the classification performance. We quantified the gain obtained throught the introduction of taxonomic and experimental information, compared to classification based on chemical information alone. We used our approach with standard machine learning models (K-nearest neighbors, random forests and deep neural networks), as well as the recently propos…
The use of machine learning (ML) methods for prediction and forecasting has become widespread across the quantitative sciences. However, there are many known methodological pitfalls, including data leakage, in ML-based science. In this paper, we systematically investigate reproducibility issues in ML-based science. We show that data leakage is indeed a widespread problem and has led to severe reproducibility failures. Specifically, through a survey of literature in research communities that adopted ML methods, we find 17 fields where errors have been found, collectively affecting 329 papers an…
University of Cambridge（剑桥大学）；Allen Institute for AI（艾伦人工智能研究所）
Amid mounting concern about the reliability and credibility of machine learning research, we present a principled framework for making robust and generalizable claims: the multiverse analysis. Our framework builds upon the multiverse analysis (Steegen et al., 2016) introduced in response to psychology's own reproducibility crisis. To efficiently explore high-dimensional and often continuous ML search spaces, we model the multiverse with a Gaussian Process surrogate and apply Bayesian experimental design. Our framework is designed to facilitate drawing robust scientific conclusions about model …
Data used to train supervised machine learning models are commonly split into independent training, validation, and test sets. This paper illustrates that complex data leakage cases have occurred in the no-reference image and video quality assessment literature. Recently, papers in several journals reported performance results well above the best in the field. However, our analysis shows that information from the test set was inappropriately used in the training process in different ways and that the claimed performance results cannot be achieved. When correcting for the data leakage, the perf…
The rapid escalation of applying Machine Learning (ML) in various domains has led to paying more attention to the quality of ML components. There is then a growth of techniques and tools aiming at improving the quality of ML components and integrating them into the ML-based system safely. Although most of these tools use bugs' lifecycle, there is no standard benchmark of bugs to assess their performance, compare them and discuss their advantages and weaknesses. In this study, we firstly investigate the reproducibility and verifiability of the bugs in ML-based systems and show the most importan…
