Although recent model-free reinforcement learning algorithms have been shown to be capable of mastering complicated decision-making tasks, the sample complexity of these methods has remained a hurdle to utilizing them in many real-world applications. In this regard, model-based reinforcement learning proposes some remedies. Yet, inherently, model-based methods are more computationally expensive and susceptible to sub-optimality. One reason is that model-generated data are always less accurate than real data, and this often leads to inaccurate transition and reward function models. With the aim to mitigate this problem, this work presents the notion of survival by discussing cases in which the agent's goal is to survive and its analogy to maximizing the expected rewards. To that end, a substitute model for the reward function approximator is introduced that learns to avoid terminal states rather than to maximize accumulated rewards from safe states. Focusing on terminal states, as a small fraction of state-space, reduces the training effort drastically. Next, a model-based reinforcement learning method is proposed (Survive) to train an agent to avoid dangerous states through a safety map model built upon temporal credit assignment in the vicinity of terminal states. Finally, the performance of the presented algorithm is investigated, along with a comparison between the proposed and current methods.
Survival analysis is playing a major role in manufacturing sector by analyzing occurrence of any unwanted event based on the input data. Predictive maintenance, which is a part of survival analysis, helps to find any device failure based on the current incoming data from different sensor or any equipment. Deep learning techniques were used to automate the predictive maintenance problem to some extent, but they are not very helpful in predicting the device failure for the input data which the algorithm had not learned. Since neural network predicts the output based on previous learned input fea…
There has been considerable interest in leveraging RL and stochastic control methods to learn optimal treatment strategies for critically ill patients, directly from observational data. However, there is significant ambiguity on the control objective and on the best reward choice for the standard RL objective. In this work, we propose a clinically motivated control objective for critically ill patients, for which the value functions have a simple medical interpretation. Further, we present theoretical results and adapt our method to a practical Deep RL algorithm, which can be used alongside an…
Obtaining a survival strategy (policy) is one of the fundamental problems of biological agents. In this paper, we generalize the formulation of previous research related to the survival of an agent and we formulate the survival problem as a maximization of the multi-step survival probability in future time steps. We introduce a method for converting the maximization of multi-step survival probability into a classical reinforcement learning problem. Using this conversion, the reward function (negative temporal cost function) is expressed as the log of the temporal survival probability. And we s…
Vilcek Institute of Graduate Biomedical Sciences, NYU Grossman School of Medicine（纽约大学格罗斯曼医学院维尔切克生物医学科学研究生院）；NYU Langone Health System（纽约大学朗格尼医疗系统）；Merck & Co., Inc（默克公司）；New York University（纽约大学）
The prognosis for patients with epithelial ovarian cancer remains dismal despite improvements in survival for other cancers. Treatment involves multiple lines of chemotherapy and becomes increasingly heterogeneous after first-line therapy. Reinforcement learning with real-world outcomes data has the potential to identify novel treatment strategies to improve overall survival. We design a reinforcement learning environment to model epithelial ovarian cancer treatment trajectories and use model free reinforcement learning to investigate therapeutic regimens for simulated patients.
Stochastic Decision Horizons for Survival-Based Constrained Reinforcement Learning
Max Planck Institute for Human Cognitive and Brain Sciences（马克斯·普朗克人类认知与脑科学研究所）；Center for Scalable Data Analytics and Artificial Intelligence (ScaDS.AI)（可扩展数据分析与人工智能中心 (ScaDS.AI)）；Hertie Institute for Clinical Brain Research & Center for Integrative Neuroscience（赫尔特临床脑研究所在线及整合神经科学中心）；University of Tübingen（图宾根大学）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
CleanSurvival: Automated data preprocessing for time-to-event models using reinforcement learning
German University in Cairo（埃及德国亚历山大大学）；German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心（DFKI））；University of Saarland（萨尔大学）；University of Kaiserslautern–Landau (RPTU)（凯撒斯劳滕-兰道大学（RPTU））
In both the fields of computer science and medicine there is very strong interest in developing personalized treatment policies for patients who have variable responses to treatments. In particular, I aim to find an optimal personalized treatment policy which is a non-deterministic function of the patient specific covariate data that maximizes the expected survival time or clinical outcome. I developed an algorithmic framework to solve multistage decision problem with a varying number of stages that are subject to censoring in which the "rewards" are expected survival times. In specific, I dev…
Solving hard-exploration environments in an important challenge in Reinforcement Learning. Several approaches have been proposed and studied, such as Intrinsic Motivation, co-evolution of agents and tasks, and multi-agent competition. In particular, the interplay between multiple agents has proven to be capable of generating human-relevant emergent behaviour that would be difficult or impossible to learn in single-agent settings. In this work, an extensible competitive environment for multi-agent interplay was developed, which features realistic physics and human-relevant semantics. Moreover, …
Counterfactual Survival Q-learning via Buckley-James Boosting, with Applications to ACTG 175 and CALGB 8923
Division of Biostatistics, The Ohio State University（俄亥俄州立大学生物统计学系）；Division of Science and Mathematics, University of Minnesota-Morris（明尼苏达州莫里斯大学科学与数学系）
Reward Engineering for Spatial Epidemic Simulations: A Reinforcement Learning Platform for Individual Behavioral Learning
Department of Computer Science University of British Columbia（计算机科学系，不列颠哥伦比亚大学）；Department of Mathematics and Institute of Applied Mathematics University of British Columbia（数学系和应用数学研究所，不列颠哥伦比亚大学）
Enhanced Evolutionary Multi-Objective Deep Reinforcement Learning for Reliable and Efficient Wireless Rechargeable Sensor Networks
College of Computer Science and Technology, Jilin University（吉林大学计算机科学与技术学院）；Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Jilin University（教育部符号计算与知识工程重点实验室）；College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算机与数据科学学院）；School of Information and Communication Engineering, Hainan University（海南大学信息与通信工程学院）；College of National Key Laboratory on Near-Surface Detection, Beijing（北京国家级近表面探测实验室）
Equitable Survival Prediction: A Fairness-Aware Survival Modeling (FASM) Approach
Duke-NUS Medical School（杜克-新加坡国立大学医学院）；Duke University School of Medicine（杜克大学医学院）；Duke AI Health（杜克人工智能健康中心）；Mass General Brigham（麻省总医院布里格姆）；Harvard Medical School（哈佛医学院）；National University of Singapore（新加坡国立大学）
NAVER R&D Center（NAVER研发中心）；NAVER CLOVA；NAVER AI LAB（NAVER人工智能实验室）
Survival analysis appears in various fields such as medicine, economics, engineering, and business. Recent studies showed that the Ordinary Differential Equation (ODE) modeling framework unifies many existing survival models while the framework is flexible and widely applicable. However, naively applying the ODE framework to survival analysis problems may model fiercely changing density function which may worsen the model's performance. Though we can apply L1 or L2 regularizers to the ODE model, their effect on the ODE modeling framework is barely known. In this paper, we propose hazard gradie…
There has been increasing interest in modeling survival data using deep learning methods in medical research. In this paper, we proposed a Bayesian hierarchical deep neural networks model for modeling and prediction of survival data. Compared with previously studied methods, the new proposal can provide not only point estimate of survival probability but also quantification of the corresponding uncertainty, which can be of crucial importance in predictive modeling and subsequent decision making. The favorable statistical properties of point and uncertainty estimates were demonstrated by simula…
Deep Survival Analysis for Competing Risk Modeling with Functional Covariates and Missing Data Imputation
Department of Quantitative Health Science, Cleveland Clinic, Cleveland OH, USA（定量健康科学系，克利夫兰诊所）；Department of Pulmonary and Critical Care Medicine, Cleveland Clinic, Cleveland OH, USA（呼吸与重症医学科，克利夫兰诊所）；Department of Statistics, Purdue University, Lafayette IN, USA（统计系，普渡大学）
University of Alberta（阿尔伯塔大学）；Alberta Machine Intelligence Institute (Amii)（阿尔伯塔机器智能研究所（Amii））
Cancer is one of the leading cause of death, worldwide. Many believe that genomic data will enable us to better predict the survival time of these patients, which will lead to better, more personalized treatment options and patient care. As standard survival prediction models have a hard time coping with the high-dimensionality of such gene expression (GE) data, many projects use some dimensionality reduction techniques to overcome this hurdle. We introduce a novel methodology, inspired by topic modeling from the natural language domain, to derive expressive features from the high-dimensional …
TripleSurv: Triplet Time-adaptive Coordinate Loss for Survival Analysis
China Mobile (Hangzhou) Information Technology Co. LTD.（中国移动（杭州）信息技术有限公司）；School of Biological Science and Medical Engineering, Beihang University（北京航空航天大学生物与医学工程学院）
This article analyzes the problem of estimating the time until an event occurs, also known as survival modeling. We observe through substantial experiments on large real-world datasets and use-cases that populations are largely heterogeneous. Sub-populations have different mean and variance in their survival rates requiring flexible models that capture heterogeneity. We leverage a classical extension of the logistic function into the survival setting to characterize unobserved heterogeneity using the beta distribution. This yields insights into the geometry of the problem as well as efficient …
Booz Allen Hamilton（博兹·艾伦·汉密尔顿公司）；University of Maryland, Baltimore County（马里兰大学巴尔的摩分校）
There has been increasing concern within the machine learning community that we are in a reproducibility crisis. As many have begun to work on this problem, all work we are aware of treat the issue of reproducibility as an intrinsic binary property: a paper is or is not reproducible. Instead, we consider modeling the reproducibility of a paper as a survival analysis problem. We argue that this perspective represents a more accurate model of the underlying meta-science question of reproducible research, and we show how a survival analysis allows us to draw new insights that better explain prior…
Survival modeling using deep learning, machine learning and statistical methods: A comparative analysis for predicting mortality after hospital admission
Duke-NUS Medical School（杜克-新加坡国立大学医学院）；Sorbonne University（索邦大学）；Stanford University（斯坦福大学）；Singapore General Hospital（新加坡中央医院）；National University of Singapore（新加坡国立大学）
Yale University（耶鲁大学）；Yale School of Medicine（耶鲁医学院）
Predictive modeling based on genomic data has gained popularity in biomedical research and clinical practice by allowing researchers and clinicians to identify biomarkers and tailor treatment decisions more efficiently. Analysis incorporating pathway information can boost discovery power and better connect new findings with biological mechanisms. In this article, we propose a general framework, Pathway-based Kernel Boosting (PKB), which incorporates clinical information and prior knowledge about pathways for prediction of binary, continuous and survival outcomes. We introduce appropriate loss …
APEX Data & Knowledge Management Lab（APEX数据与知识管理实验室）；Shanghai Jiao Tong University（上海交通大学）
Survival analysis is a hotspot in statistical research for modeling time-to-event information with data censorship handling, which has been widely used in many applications such as clinical research, information system and other fields with survivorship bias. Many works have been proposed for survival analysis ranging from traditional statistic methods to machine learning models. However, the existing methodologies either utilize counting-based statistics on the segmented data, or have a pre-assumption on the event probability distribution w.r.t. time. Moreover, few works consider sequential p…
Functional Decomposition and Shapley Interactions for Interpreting Survival Models
Leibniz Institute for Prevention Research（预防研究与流行病学研究所）；Faculty of Mathematics（数学系）；Computer Science, University of Bremen（计算机科学，不莱梅大学）；University of Warsaw（华沙大学）；Centre for Credible AI, Warsaw University of Technology（可信人工智能中心，华沙技术大学）；Bielefeld University（比勒菲尔德大学）
