Offline RL methods have been shown to reduce the need for environment interaction by training agents using offline collected episodes. However, these methods typically require action information to be logged during data collection, which can be difficult or even impossible in some practical cases. In this paper, we investigate the potential of using action-free offline datasets to improve online reinforcement learning, name this problem Reinforcement Learning with Action-Free Offline Pretraining (AFP-RL). We introduce Action-Free Guide (AF-Guide), a method that guides online training by extracting knowledge from action-free offline datasets. AF-Guide consists of an Action-Free Decision Transformer (AFDT) implementing a variant of Upside-Down Reinforcement Learning. It learns to plan the next states from the offline dataset, and a Guided Soft Actor-Critic (Guided SAC) that learns online with guidance from AFDT. Experimental results show that AF-Guide can improve sample efficiency and performance in online training thanks to the knowledge from the action-free offline dataset. Code is available at https://github.com/Vision-CAIR/AF-Guide.
The recent success of supervised learning methods on ever larger offline datasets has spurred interest in the reinforcement learning (RL) field to investigate whether the same paradigms can be translated to RL algorithms. This research area, known as offline RL, has largely focused on offline policy optimization, aiming to find a return-maximizing policy exclusively from offline data. In this paper, we consider a slightly different approach to incorporating offline data into sequential decision-making. We aim to answer the question, what unsupervised objectives applied to offline datasets are …
Scaling Offline Model-Based RL via Jointly-Optimized World-Action Model Pretraining
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences（多模态人工智能系统国家重点实验室，自动化研究所，中国科学院）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Alibaba Group（阿里巴巴集团）
A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning
The University of Sydney（悉尼大学）；SenseTime Computer Vision Group（商汤科技计算机视觉组）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；CUHK MMLab（香港中文大学多媒体实验室）；Institute for AI, Peking University（北京大学人工智能研究院）
Efficient Reinforcement Learning by Guiding World Models with Non-Curated Data
Aalto University（阿alto大学）；University of Edinburgh（爱丁堡大学）；ELLIS Institute Finland（芬兰ELLIS研究所）；Deep Render；Imperial College London（伦敦帝国理工学院）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；CIFAR AI Chair（CIFAR人工智能主席）；University of Alberta（阿尔伯塔大学）；Alberta Machine Intelligence Institute (Amii)（阿尔伯塔机器智能研究所（Amii））；University of Oulu（奥卢大学）
University of California, Berkeley（加州大学伯克利分校）；University of Illinois, Chicago（伊利诺伊大学芝加哥分校）
Our team is proposing to run a full-scale energy demand response experiment in an office building. Although this is an exciting endeavor which will provide value to the community, collecting training data for the reinforcement learning agent is costly and will be limited. In this work, we examine how offline training can be leveraged to minimize data costs (accelerate convergence) and program implementation costs. We present two approaches to doing so: pretraining our model to warm start the experiment with simulated tasks, and using a planning model trained to simulate the real world's reward…
Robo-ValueRL: Reliable Value Estimation for Offline-to-Online Reinforcement Learning
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；Beijing Innovation Center of Humanoid Robotics（北京人形机器人创新中心）；Beijing Forestry University（北京林业大学）；Peking University（北京大学）；Microsoft Research（微软研究院）
University of Chinese Academy of Sciences（中国科学院大学）；Peking University（北京大学）；University College London（伦敦大学学院）；Chinese Academy of Sciences（中国科学院）；King’s College London（伦敦国王学院）
The pretrain-finetuning paradigm in large-scale sequence models has made significant progress in natural language processing and computer vision tasks. However, such a paradigm is still hindered by several challenges in Reinforcement Learning (RL), including the lack of self-supervised pretraining algorithms based on offline data and efficient fine-tuning/prompt-tuning over unseen downstream tasks. In this work, we explore how prompts can improve sequence modeling-based offline reinforcement learning (offline-RL) algorithms. Firstly, we propose prompt tuning for offline RL, where a context vec…
Offline reinforcement learning (RL) allows for the training of competent agents from offline datasets without any interaction with the environment. Online finetuning of such offline models can further improve performance. But how should we ideally finetune agents obtained from offline RL training? While offline RL algorithms can in principle be used for finetuning, in practice, their online performance improves slowly. In contrast, we show that it is possible to use standard online off-policy algorithms for faster improvement. However, we find this approach may suffer from policy collapse, whe…
Reward function is essential in reinforcement learning (RL), serving as the guiding signal to incentivize agents to solve given tasks, however, is also notoriously difficult to design. In many cases, only imperfect rewards are available, which inflicts substantial performance loss for RL agents. In this study, we propose a unified offline policy optimization approach, \textit{RGM (Reward Gap Minimization)}, which can smartly handle diverse types of imperfect rewards. RGM is formulated as a bi-level optimization problem: the upper layer optimizes a reward correction term that performs visitatio…
Meta AI Research（Meta人工智能研究院）；University of California, Berkeley（加州大学伯克利分校）；University of California, Los Angeles（加州大学洛杉矶分校）
Recent work has shown that offline reinforcement learning (RL) can be formulated as a sequence modeling problem (Chen et al., 2021; Janner et al., 2021) and solved via approaches similar to large-scale language modeling. However, any practical instantiation of RL also involves an online component, where policies pretrained on passive offline datasets are finetuned via taskspecific interactions with the environment. We propose Online Decision Transformers (ODT), an RL algorithm based on sequence modeling that blends offline pretraining with online finetuning in a unified framework. Our framewor…
GOPlan: Goal-conditioned Offline Reinforcement Learning by Planning with Learned Models
University of Warwick（华威大学）；Hong Kong University of Science and Technology（香港科技大学）；Tsinghua University（清华大学）；University of Cambridge（剑桥大学）；University of Liverpool（利物浦大学）
Policy-Based Trajectory Clustering in Offline Reinforcement Learning
Institute for Interdisciplinary Information Sciences（交叉信息科学研究所）；Tsinghua University（清华大学）；Paul G. Allen School of Computer Science & Engineering（保罗·G·艾伦计算机科学与工程学院）；University of Washington（华盛顿大学）
Université de Montréal（蒙特利尔大学）；Mila（米拉人工智能研究所）；HEC Montréal（蒙特利尔高等商学院）；CIFAR（加拿大高等研究院）；Microsoft Research（微软研究院）
Data efficiency is a key challenge for deep reinforcement learning. We address this problem by using unlabeled data to pretrain an encoder which is then finetuned on a small amount of task-specific data. To encourage learning representations which capture diverse aspects of the underlying MDP, we employ a combination of latent dynamics modelling and unsupervised goal-conditioned RL. When limited to 100k steps of interaction on Atari games (equivalent to two hours of human experience), our approach significantly surpasses prior work combining offline representation pretraining with task-specifi…
Institute of Automation, CAS（中国科学院自动化研究所）；School of Artificial Intelligence, UCAS（中国科学院大学人工智能学院）；Shanghai Jiao Tong University（上海交通大学）；King’s College London（伦敦国王学院）；University College London（伦敦大学学院）
Offline reinforcement learning leverages previously-collected offline datasets to learn optimal policies with no necessity to access the real environment. Such a paradigm is also desirable for multi-agent reinforcement learning (MARL) tasks, given the increased interactions among agents and with the enviroment. Yet, in MARL, the paradigm of offline pre-training with online fine-tuning has not been studied, nor datasets or benchmarks for offline MARL research are available. In this paper, we facilitate the research by providing large-scale datasets, and use them to examine the usage of the Deci…
Uni-O4: Unifying Online and Offline Deep Reinforcement Learning with Multi-Step On-Policy Optimization
Shanghai Qi Zhi Institute（上海期智研究院）；Tsinghua University（清华大学）；Shanghai AI Lab（上海人工智能实验室）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
QPO: Query-dependent Prompt Optimization via Multi-Loop Offline Reinforcement Learning
Tsinghua University（清华大学）；SenseTime Research（商汤科技研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Sun Yat-Sen University（中山大学）；Nanyang Technological University（南洋理工大学）
Scaling World-Model Reinforcement Learning Through Diffusion Policy Optimization
Dynamic Systems Lab, University College London（伦敦大学学院动态系统实验室）；College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算与数据科学学院）；School of Intelligence Science and Technology, Peking University（北京大学智能科学与技术学院）；Santa Fe Institute（圣塔菲研究所）；School of Statistics and Data Science, Shanghai University of Finance and Economics（上海财经大学统计与数据科学学院）；Department of Computing, Imperial College London（伦敦帝国理工学院计算系）
Reward-free Pretraining for Reinforcement Learning via Occupancy Coverage Maximization
Computational Statistics and Machine Learning - Istituto Italiano di Tecnologia（计算统计与机器学习 - 意大利技术研究院）；AI Centre, Computer Science Department, University College London（人工智能中心，计算机科学系，伦敦大学学院）
Unsupervised pretraining and dropout have been well studied, especially with respect to regularization and output consistency. However, our understanding about the explicit convergence rates of the parameter estimates, and their dependence on the learning (like denoising and dropout rate) and structural (like depth and layer lengths) aspects of the network is less mature. An interesting question in this context is to ask if the network structure could "guide" the choices of such learning parameters. In this work, we explore these gaps between network structure, the learning mechanisms and thei…
