Offline reinforcement learning (RL) learns policies entirely from static datasets, thereby avoiding the challenges associated with online data collection. Practical applications of offline RL will inevitably require learning from datasets where the variability of demonstrated behaviors changes non-uniformly across the state space. For example, at a red light, nearly all human drivers behave similarly by stopping, but when merging onto a highway, some drivers merge quickly, efficiently, and safely, while many hesitate or merge dangerously. Both theoretically and empirically, we show that typical offline RL methods, which are based on distribution constraints fail to learn from data with such non-uniform variability, due to the requirement to stay close to the behavior policy to the same extent across the state space. Ideally, the learned policy should be free to choose per state how closely to follow the behavior policy to maximize long-term return, as long as the learned policy stays within the support of the behavior policy. To instantiate this principle, we reweight the data distribution in conservative Q-learning (CQL) to obtain an approximate support constraint formulation. The reweighted distribution is a mixture of the current policy and an additional policy trained to mine poor actions that are likely under the behavior policy. Our method, CQL (ReDS), is simple, theoretically motivated, and improves performance across a wide range of offline RL problems in Atari games, navigation, and pixel-based manipulation.
RORL: Robust Offline Reinforcement Learning via Conservative Smoothing
Hong Kong University of Science and Technology(香港科技大学);Tsinghua University(清华大学);Northwestern University(西北大学);Tencent Robotics X(腾讯Robotics X实验室);Shanghai AI Laboratory(上海人工智能实验室)
Beyond Non-Expert Demonstrations: Outcome-Driven Action Constraint for Offline Reinforcement Learning
College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics(南京航空航天大学计算机科学与技术学院);School of Computer and Information Technology, Shanxi University(山西大学计算机与信息技术学院)
DAWM: Diffusion Action World Models for Offline Reinforcement Learning via Action-Inferred Transitions
Department of Computer Science, University of Munich, Munich, Germany(慕尼黑大学计算机科学系);Munich Center for Machine Learning (MCML), Munich, Germany(慕尼黑机器学习中心(MCML))
Safe Deployment of Offline Reinforcement Learning via Input Convex Action Correction
Department of Chemical Engineering, Imperial College London(帝国理工学院化学工程系);Shell Information Technology International BV(壳牌信息科技国际有限公司);Shell Global Solutions International BV(壳牌全球解决方案国际有限公司)
ChiPFormer: Transferable Chip Placement via Offline Decision Transformer
The University of Hong Kong(香港大学);Shanghai AI Laboratory(上海人工智能实验室);Huawei Noah’s Ark Lab(华为诺亚方舟实验室);Zhejiang University(浙江大学);Tianjin University(天津大学)
Solving Continual Offline RL through Selective Weights Activation on Aligned Spaces
School of Artificial Intelligence, Jilin University(吉林大学人工智能学院);School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University(中山大学深圳校区网络科学与技术学院);MoE Key Lab of Artificial Intelligence, Shanghai Jiao Tong University(教育部人工智能重点实验室);Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);College of Computing and Data Science, Nanyang Technological University(南洋理工大学计算与数据科学学院);Computer Science and Engineering, Lehigh University(利哈伊大学计算机科学与工程学院)
Coordinating Ride-Pooling with Public Transit using Reward-Guided Conservative Q-Learning: An Offline Training and Online Fine-Tuning Reinforcement Learning Framework
The Hong Kong University of Science and Technology(香港科技大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州))
Counterfactual Learning of Stochastic Policies with Continuous Actions
Criteo AI Lab(Criteo AI实验室);Univ. Grenoble Alpes(格勒诺布尔大学);Inria(法国国家信息与自动化研究所);CNRS(法国国家科学研究中心);Grenoble INP(格勒诺布尔理工学院);LJK(约瑟夫·傅里叶大学实验室(注:LJK为联合机构,按要求保留英文原名));Center for Data Science, New York University(纽约大学数据科学中心)
Data-driven simulator of multi-animal behavior with unknown dynamics via offline and online reinforcement learning
Graduate School of Informatics, Nagoya University, Japan(名古屋大学信息学研究科);RIKEN Center for Advanced Intelligence Project, Japan(RIKEN高级智能项目研究中心);Graduate School of Arts and Sciences, The University of Tokyo, Japan(东京大学文学系研究科);Project team for SIP, Japan(SIP项目团队);Japan Agency for Marine-Earth Science and Technology, Japan(日本海洋地球科学技术机构);Faculty of Education, Shitennoji University, Japan(世田谷大学教育学部);Graduate School of Engineering, The University of Tokyo, Japan(东京大学工学研究科);Graduate School of Science and Technology, Niigata University, Japan(新潟大学科学技术研究科);Graduate School of Science, Nagoya University, Japan(名古屋大学理学研究科);Principles of Informatics Research Division, National Institute of Informatics, Japan(信息学原理研究部门,日本信息处理技术研究所);Graduate School of Information Science, The University of Osaka, Japan(大阪大学信息科学研究科)
Disentangling Policy from Offline Task Representation Learning via Adversarial Data Augmentation
National Key Laboratory for Novel Software Technology(计算机软件新技术国家重点实验室);School of Artificial Intelligence(人工智能学院);Nanjing University(南京大学);Polixir Technologies(铂乐奇科技)
Advancing Offline Handwritten Text Recognition: A Systematic Review of Data Augmentation and Generation Techniques
Artificial Intelligence and Innovation Centre, University of Kurdistan Hewler(库尔德斯坦胡勒尔大学人工智能与创新中心);Computer Science and Engineering Department, School of Science and Engineering, University of Kurdistan Hewler(库尔德斯坦胡勒尔大学科学与工程学院计算机科学与工程系);School of Computing and Engineering, University of Huddersfield(赫德斯菲尔德大学计算与工程学院);Department of Computer Science, College of Science, Charmo University(查莫大学科学学院计算机科学系);Software and Informatics Engineering, College of Engineering, Salahaddin University-Erbil(萨拉赫丁大学埃尔比尔工程学院软件与信息技术工程)
Time to Split: Exploring Data Splitting Strategies for Offline Evaluation of Sequential Recommenders
AIRI, Skoltech Moscow Russian Federation(AIRI,斯克里普钦斯基因工学院莫斯科俄罗斯联邦);Sber AI Lab, Skoltech Moscow Russian Federation(Sber AI Lab,斯克里普钦斯基因工学院莫斯科俄罗斯联邦);Sber AI Lab Moscow Russian Federation(Sber AI Lab,莫斯科斯克里普钦斯基因工学院俄罗斯联邦);Sber AI Lab, HSE University Moscow Russian Federation(Sber AI Lab,莫斯科高等经济大学俄罗斯联邦)