Deep reinforcement learning (RL) works impressively in some environments and fails catastrophically in others. Ideally, RL theory should be able to provide an understanding of why this is, i.e. bounds predictive of practical performance. Unfortunately, current theory does not quite have this ability. We compare standard deep RL algorithms to prior sample complexity bounds by introducing a new dataset, BRIDGE. It consists of 155 deterministic MDPs from common deep RL benchmarks, along with their corresponding tabular representations, which enables us to exactly compute instance-dependent bounds. We choose to focus on deterministic environments because they share many interesting properties of stochastic environments, but are easier to analyze. Using BRIDGE, we find that prior bounds do not correlate well with when deep RL succeeds vs. fails, but discover a surprising property that does. When actions with the highest Q-values under the random policy also have the highest Q-values under the optimal policy (i.e. when it is optimal to be greedy on the random policy's Q function), deep RL tends to succeed; when they don't, deep RL tends to fail. We generalize this property into a new complexity measure of an MDP that we call the effective horizon, which roughly corresponds to how many steps of lookahead search would be needed in that MDP in order to identify the next optimal action, when leaf nodes are evaluated with random rollouts. Using BRIDGE, we show that the effective horizon-based bounds are more closely reflective of the empirical performance of PPO and DQN than prior sample complexity bounds across four metrics. We also find that, unlike existing bounds, the effective horizon can predict the effects of using reward shaping or a pre-trained exploration policy. Our code and data are available at https://github.com/cassidylaidlaw/effective-horizon
Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint
University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Salesforce AI Research(Salesforce AI研究院);The Hong Kong University of Science and Technology(香港科技大学)
DHRL: A Graph-Based Approach for Long-Horizon and Sparse Hierarchical Reinforcement Learning
Seoul National University(首尔国立大学);Artificial Intelligence Institute of Seoul National University (AIIS)(首尔国立大学人工智能研究所(AIIS));Automation and Systems Research Institute (ASRI)(自动化与系统研究所(ASRI))
Bridging AI and Clinical Practice: Integrating Automated Sleep Scoring Algorithm with Uncertainty-Guided Physician Review
University of Bern(伯尔尼大学);University of Applied Sciences and Arts of Southern Switzerland(瑞士南部应用科学与艺术大学);Inselspital, Bern University Hospital(伯尔尼大学医院);Ohio Sleep Medicine Institute(俄亥俄睡眠医学研究所)
When Are Scoring Rules Proper? Bridging Theory and Practice in Survival Model Evaluation
Department of Cancer Genetics, Institute for Cancer Research, Oslo University Hospital(癌症遗传学系,癌症研究所,奥斯陆大学医院);Department of Biostatistics, Faculty of Medicine, Oslo Centre for Biostatistics and Epidemiology (OCBE), University of Oslo, Norway(生物统计学系,医学学院,奥斯陆生物统计与流行病学中心(OCBE),奥斯陆大学,挪威);OSPO Now, London, UK(伦敦,英国OSPO Now);Department of Mathematics, Faculty of Mathematics and Natural Sciences, University of Oslo, Norway(数学系,数学与自然科学学院,奥斯陆大学,挪威);Department of Statistics, LMU Munich, Germany(统计学系,慕尼黑大学);Munich Center for Machine Learning (MCML), LMU Munich, Germany(慕尼黑机器学习中心(MCML),慕尼黑大学,德国);Statistical Consulting Unit (StaBLab), LMU Munich, Germany(统计咨询单位(StaBLab),慕尼黑大学,德国);Leibniz Institute for Prevention Research and Epidemiology - BIPS, Bremen, Germany(预防研究与流行病学研究所—BIPS,不莱梅,德国);Faculty of Mathematics and Computer Science, University of Bremen, Germany(数学与计算机科学学院,不莱梅大学,德国)