Learning a good representation is an essential component for deep reinforcement learning (RL). Representation learning is especially important in multitask and partially observable settings where building a representation of the unknown environment is crucial to solve the tasks. Here we introduce Prediction of Bootstrap Latents (PBL), a simple and flexible self-supervised representation learning algorithm for multitask deep RL. PBL builds on multistep predictive representations of future observations, and focuses on capturing structured information about environment dynamics. Specifically, PBL trains its representation by predicting latent embeddings of future observations. These latent embeddings are themselves trained to be predictive of the aforementioned representations. These predictions form a bootstrapping effect, allowing the agent to learn more about the key aspects of the environment dynamics. In addition, by defining prediction tasks completely in latent space, PBL provides the flexibility of using multimodal observations involving pixel images, language instructions, rewards and more. We show in our experiments that PBL delivers across-the-board improved performance over state of the art deep RL agents in the DMLab-30 and Atari-57 multitask setting.
Representation Learning Enables Scalable Multitask Deep Reinforcement Learning
Mila – Québec AI Institute（魁北克AI研究所）；Université de Montréal（蒙特利尔大学）；McGill University（麦吉尔大学）；CIFAR AI Chair（CIFAR人工智能 chair）；Google DeepMind（谷歌DeepMind）
Mila - Quebec AI Institute（米拉-魁北克人工智能研究所）；McGill University（麦吉尔大学）；CIFAR（加拿大高等研究院）；Meta AI Research（Meta人工智能研究院）
Estimating value functions is a core component of reinforcement learning algorithms. Temporal difference (TD) learning algorithms use bootstrapping, i.e. they update the value function toward a learning target using value estimates at subsequent time-steps. Alternatively, the value function can be updated toward a learning target constructed by separately predicting successor features (SF)--a policy-dependent model--and linearly combining them with instantaneous rewards. We focus on bootstrapping targets used when estimating value functions, and propose a new backup target, the $η$-return mixt…
University of Alberta（阿尔伯塔大学）；University of Toronto（多伦多大学）；Huawei Technologies Canada, Ltd.（加拿大华为技术有限公司）
We explore fixed-horizon temporal difference (TD) methods, reinforcement learning algorithms for a new kind of value function that predicts the sum of rewards over a $\textit{fixed}$ number of future time steps. To learn the value function for horizon $h$, these algorithms bootstrap from the value function for horizon $h-1$, or some shorter horizon. Because no value function bootstraps from itself, fixed-horizon methods are immune to the stability problems that plague other off-policy TD methods using function approximation (also known as "the deadly triad"). Although fixed-horizon methods req…
Scalable predictive processing framework for multitask caregiving robots
National Institute of Neuroscience, National Center of Neurology and Psychiatry（国立精神·神经医疗研究中心 神经科学研究所）；Future Robotics Organization, Waseda University（早稻田大学未来机器人机构）；Department of Intermedia Art and Science, Waseda University（早稻田大学跨媒体艺术与科学系）
Genomics data such as RNA gene expression, methylation and micro RNA expression are valuable sources of information for various clinical predictive tasks. For example, predicting survival outcomes, cancer histology type and other patients' related information is possible using not only clinical data but molecular data as well. Moreover, using these data sources together, for example in multitask learning, can boost the performance. However, in practice, there are many missing data points which leads to significantly lower patient numbers when analysing full cases, which in our setting refers t…
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Tencent（腾讯）
The exploration/exploitation (E&E) dilemma lies at the core of interactive systems such as online advertising, for which contextual bandit algorithms have been proposed. Bayesian approaches provide guided exploration with principled uncertainty estimation, but the applicability is often limited due to over-simplified assumptions. Non-Bayesian bootstrap methods, on the other hand, can apply to complex problems by using deep reward models, but lacks clear guidance to the exploration behavior. It still remains largely unsolved to develop a practical method for complex deep contextual bandits. In …
Learning Stateful Predictive Knowledge From Experience
University College London（伦敦大学学院）；National University of Singapore（新加坡国立大学）；Chinese Academy of Sciences（中国科学院）；Brown University（布朗大学）；University of Bristol（布里斯托尔大学）；University of Edinburgh（爱丁堡大学）；Zhongguancun Academy（中关村科学院）；The Yangtze River Delta（长三角）
Machine learning approaches have been effective in predicting adverse outcomes in different clinical settings. These models are often developed and evaluated on datasets with heterogeneous patient populations. However, good predictive performance on the aggregate population does not imply good performance for specific groups. In this work, we present a two-step framework to 1) learn relevant patient subgroups, and 2) predict an outcome for separate patient populations in a multi-task framework, where each population is a separate task. We demonstrate how to discover relevant groups in an unsup…
Multitask-Informed Prior for In-Context Learning on Tabular Data: Application to Steel Property Prediction
Dep. of Electrical and Computer Engineering, McGill University（麦吉尔大学电气与计算机工程系）；Mila – Québec AI Institute（魁北克AI研究所）；Dep. of Mining and Materials Engineering, McGill University（麦吉尔大学采矿与材料工程系）；Algoma Steel Inc.（阿尔戈马钢铁公司）
University of Eastern Finland（东芬兰大学）；Tampere University（坦佩雷大学）
The Alzheimer's Disease Assessment Scale-Cognitive subscale (ADAS-Cog) is a neuropsychological tool that has been designed to assess the severity of cognitive symptoms of dementia. Personalized prediction of the changes in ADAS-Cog scores could help in timing therapeutic interventions in dementia and at-risk populations. In the present work, we compared single and multitask learning approaches to predict the changes in ADAS-Cog scores based on T1-weighted anatomical magnetic resonance imaging (MRI). In contrast to most machine learning-based prediction methods ADAS-Cog changes, we stratified t…
We introduce Mix&Match (M&M) - a training framework designed to facilitate rapid and effective learning in RL agents, especially those that would be too slow or too challenging to train otherwise. The key innovation is a procedure that allows us to automatically form a curriculum over agents. Through such a curriculum we can progressively train more complex agents by, effectively, bootstrapping from solutions found by simpler agents. In contradistinction to typical curriculum learning approaches, we do not gradually modify the tasks or environments presented, but instead use a process to gradu…
University of Macau（澳门大学）；Central South University（中南大学）
Background: Pharmacokinetic evaluation is one of the key processes in drug discovery and development. However, current absorption, distribution, metabolism, excretion prediction models still have limited accuracy. Aim: This study aims to construct an integrated transfer learning and multitask learning approach for developing quantitative structure-activity relationship models to predict four human pharmacokinetic parameters. Methods: A pharmacokinetic dataset included 1104 U.S. FDA approved small molecule drugs. The dataset included four human pharmacokinetic parameter subsets (oral bioavailab…
With the growing popularity of wearable devices, the ability to utilize physiological data collected from these devices to predict the wearer's mental state such as mood and stress suggests great clinical applications, yet such a task is extremely challenging. In this paper, we present a general platform for personalized predictive modeling of behavioural states like students' level of stress. Through the use of Auto-encoders and Multitask learning we extend the prediction of stress to both sequences of passive sensor data and high-level covariates. Our model outperforms the state-of-the-art i…
University of Crete（克里特大学）；Gnosis Data Analysis PC（Gnosis 数据分析公司）
Cross-Validation (CV), and out-of-sample performance-estimation protocols in general, are often employed both for (a) selecting the optimal combination of algorithms and values of hyper-parameters (called a configuration) for producing the final predictive model, and (b) estimating the predictive performance of the final model. However, the cross-validated performance of the best configuration is optimistically biased. We present an efficient bootstrap method that corrects for the bias, called Bootstrap Bias Corrected CV (BBC-CV). BBC-CV's main idea is to bootstrap the whole process of selecti…
When there is a distributional shift between data used to train a predictive algorithm and current data, performance can suffer. This is known as the domain adaptation problem. Bootstrap aggregating, or bagging, is a popular method for improving stability of predictive algorithms, while reducing variance and protecting against over-fitting. This article proposes a domain adaptive bagging method coupled with a new iterative nearest neighbor sampler. The key idea is to draw bootstrap samples from the training data in such a manner that their distribution equals that of new testing data. The prop…
CrunchLLM: Multitask LLMs for Structured Business Reasoning and Outcome Prediction
Department of Computer Science University of Kentucky（计算机科学系 俄亥俄大学）；Department of Computer Science, Institute for Biomedical Informatics University of Kentucky（计算机科学系 生物医学信息学研究所 俄亥俄大学）
We validate the recently introduced deep learning classification adapted Delta method by a comparison with the classical Bootstrap. We show that there is a strong linear relationship between the quantified predictive epistemic uncertainty levels obtained from the two methods when applied on two LeNet-based neural network classifiers using the MNIST and CIFAR-10 datasets. Furthermore, we demonstrate that the Delta method offers a five times computation time reduction compared to the Bootstrap.
FlexCare: Leveraging Cross-Task Synergy for Flexible Multimodal Healthcare Prediction
Institute of Information Science, Beijing Jiaotong University（北京交通大学信息科学研究所）；Medical Big Data Research Center, Chinese PLA General Hospital（中国人民解放军总医院医学大数据中心）
From Alignment to Prediction: A Study of Self-Supervised Learning and Predictive Representation Learning
Department of Information and Communication Technology, School of Technology, Pandit Deendayal Energy University, Gandhinagar, Gujarat 382007, India（信息与通信技术系，技术学院，潘迪·德恩达尔能源大学，古吉拉特邦加尔各纳，382007，印度）
Reducing cross-sample prediction churn in scientific machine learning
Helmholtz-Zentrum Berlin für Materialien und Energie GmbH（柏林材料与能源研究中心）；Helmholtz Institute for Polymers in Energy Applications Jena（耶拿聚合物能源应用赫尔姆霍茨研究所）；Laboratory of Organic and Macromolecular Chemistry (IOMC)（有机与大分子化学实验室）；Center for Energy and Environmental Chemistry Jena（耶拿能源与环境化学中心）；Jena Center for Soft Matter (JCSM)（耶拿软物质研究中心）
This paper presents an alternative approach to p-values in regression settings. This approach, whose origins can be traced to machine learning, is based on the leave-one-out bootstrap for prediction error. In machine learning this is called the out-of-bag (OOB) error. To obtain the OOB error for a model, one draws a bootstrap sample and fits the model to the in-sample data. The out-of-sample prediction error for the model is obtained by calculating the prediction error for the model using the out-of-sample data. Repeating and averaging yields the OOB error, which represents a robust cross-vali…
Recently, artificial intelligence technologies and algorithms have become a major focus for advancements in treatment planning for radiation therapy. As these are starting to become incorporated into the clinical workflow, a major concern from clinicians is not whether the model is accurate, but whether the model can express to a human operator when it does not know if its answer is correct. We propose to use Monte Carlo dropout (MCDO) and the bootstrap aggregation (bagging) technique on deep learning models to produce uncertainty estimations for radiation therapy dose prediction. We show that…
Machine learning models $-$ now commonly developed to screen, diagnose, or predict health conditions $-$ are evaluated with a variety of performance metrics. An important first step in assessing the practical utility of a model is to evaluate its average performance over an entire population of interest. In many settings, it is also critical that the model makes good predictions within predefined subpopulations. For instance, showing that a model is fair or equitable requires evaluating the model's performance in different demographic subgroups. However, subpopulation performance metrics are t…
We consider the problem in regression analysis of identifying subpopulations that exhibit different patterns of response, where each subpopulation requires a different underlying model. Unlike statistical cohorts, these subpopulations are not known a priori; thus, we refer to them as cadres. When the cadres and their associated models are interpretable, modeling leads to insights about the subpopulations and their associations with the regression target. We introduce a discriminative model that simultaneously learns cadre assignment and target-prediction rules. Sparsity-inducing priors are pla…
