There has been increasing concern within the machine learning community that we are in a reproducibility crisis. As many have begun to work on this problem, all work we are aware of treat the issue of reproducibility as an intrinsic binary property: a paper is or is not reproducible. Instead, we consider modeling the reproducibility of a paper as a survival analysis problem. We argue that this perspective represents a more accurate model of the underlying meta-science question of reproducible research, and we show how a survival analysis allows us to draw new insights that better explain prior longitudinal data. The data and code can be found at https://github.com/EdwardRaff/Research-Reproducibility-Survival-Analysis
When modelling competing risks survival data, several techniques have been proposed in both the statistical and machine learning literature. State-of-the-art methods have extended classical approaches with more flexible assumptions that can improve predictive performance, allow high dimensional data and missing values, among others. Despite this, modern approaches have not been widely employed in applied settings. This article aims to aid the uptake of such methods by providing a condensed compendium of competing risks survival methods with a unified notation and interpretation across approach…
Maximizing product use is a central goal of many businesses, which makes retention and monetization two central analytics metrics in games. Player retention may refer to various duration variables quantifying product use: total playtime or session playtime are popular research targets, and active playtime is well-suited for subscription games. Such research often has the goal of increasing player retention or conversely decreasing player churn. Survival analysis is a framework of powerful tools well suited for retention type data. This paper contributes new methods to game analytics on how pla…
PISA: An AI Pipeline for Interpretable-by-design Survival Analysis Providing Multiple Complexity-Accuracy Trade-off Models
a Leiden University Medical Center, Leiden, The Netherlands；b Amsterdam University Medical Centers location Vrije Universiteit Amsterdam, Amsterdam, The Netherlands；c Cancer Center Amsterdam, Cancer Treatment；Quality of Life, Amsterdam, The Netherlands；d Centrum Wiskunde \& Informatica, Amsterdam, The Netherlands
Interpretable Survival Analysis for Heart Failure Risk Prediction
Stanford University（斯坦福大学）；VA Palo Alto Health Care System（帕洛阿尔托退伍军人医疗保健系统）；Kaiser Permanente Northern California Division of Research（凯撒医疗北加州研究部）；Stanford Medicine（斯坦福医学院）
Palliative medicine is an interdisciplinary specialty focusing on improving quality of life (QOL) for patients with serious illness and their families. Palliative care programs are available or under development at over 80% of large US hospitals (300+ beds). Palliative care clinical trials present unique analytic challenges relative to evaluating the palliative care treatment efficacy which is to improve patients diminishing QOL as disease progresses towards end of life (EOL). A unique feature of palliative care clinical trials is that patients will experience decreasing QOL during the trial d…
There has been increasing interest in modeling survival data using deep learning methods in medical research. In this paper, we proposed a Bayesian hierarchical deep neural networks model for modeling and prediction of survival data. Compared with previously studied methods, the new proposal can provide not only point estimate of survival probability but also quantification of the corresponding uncertainty, which can be of crucial importance in predictive modeling and subsequent decision making. The favorable statistical properties of point and uncertainty estimates were demonstrated by simula…
Survival data is encountered in a range of disciplines, most notably health and medical research. Although Bayesian approaches to the analysis of survival data can provide a number of benefits, they are less widely used than classical (e.g. likelihood-based) approaches. This may be in part due to a relative absence of user-friendly implementations of Bayesian survival models. In this article we describe how the rstanarm R package can be used to fit a wide range of Bayesian survival models. The rstanarm package facilitates Bayesian regression modelling by providing a user-friendly interface (us…
McGill University（麦吉尔大学）；Facebook AI Research（Facebook人工智能研究院）；CIFAR（加拿大高等研究院）；Université de Montréal（蒙特利尔大学）；Yahoo! Research（雅虎研究院）；Télécom Paris（巴黎电信学院）；Institut Polytechnique de France（法国理工学院）；University of Washington（华盛顿大学）；Apple（苹果公司）；Google（谷歌）
One of the challenges in machine learning research is to ensure that presented and published results are sound and reliable. Reproducibility, that is obtaining similar results as presented in a paper or talk, using the same code and data (when available), is a necessary step to verify the reliability of research findings. Reproducibility is also an important step to promote open and accessible research, thereby allowing the scientific community to quickly integrate new findings and convert ideas to practice. Reproducibility also promotes the use of robust experimental workflows, which potentia…
Sorbonne Université（索邦大学）；Institut du Cerveau - Paris Brain Institute - ICM（巴黎脑科学院）；CNRS（法国国家科学研究中心）；Inria（法国国家信息与自动化研究所）；Inserm（法国国家健康与医学研究院）；AP-HP（巴黎公共医疗救助机构）；Hôpital de la Pitié-Salpêtrière（巴黎硝石库慈善医院）；University of Twente（特文特大学）
Reproducibility is a cornerstone of science, as the replication of findings is the process through which they become knowledge. It is widely considered that many fields of science are undergoing a reproducibility crisis. This has led to the publications of various guidelines in order to improve research reproducibility. This didactic chapter intends at being an introduction to reproducibility for researchers in the field of machine learning for medical imaging. We first distinguish between different types of reproducibility. For each of them, we aim at defining it, at describing the requiremen…
In this paper we describe a novel implementation of adaboost for prediction of survival function. We take different variations of the algorithm and compare the algorithms based on system run time and root mean square error. Our construction includes right censoring data and competing risk data too. We take different data set to illustrate the performance of the algorithms.
Massachusetts Institute of Technology（麻省理工学院）；University of Toronto（多伦多大学）；Evidation Health, Inc.（Evidation健康公司）；New York University（纽约大学）；Vector Institute（向量研究所）
Machine learning algorithms designed to characterize, monitor, and intervene on human health (ML4H) are expected to perform safely and reliably when operating at scale, potentially outside strict human supervision. This requirement warrants a stricter attention to issues of reproducibility than other fields of machine learning. In this work, we conduct a systematic evaluation of over 100 recently published ML4H research papers along several dimensions related to reproducibility. We find that the field of ML4H compares poorly to more established machine learning fields, particularly concerning …
Delft University of Technology（代尔夫特理工大学）；Leiden University Medical Center（莱顿大学医学中心）；University of Copenhagen（哥本哈根大学）
In this paper, we discuss the approaches we took and trade-offs involved in making a paper on a conceptual topic in pattern recognition research fully reproducible. We discuss our definition of reproducibility, the tools used, how the analysis was set up, show some examples of alternative analyses the code enables and discuss our views on reproducibility.
