We study the problem of safe offline reinforcement learning (RL), the goal is to learn a policy that maximizes long-term reward while satisfying safety constraints given only offline data, without further interaction with the environment. This problem is more appealing for real world RL applications, in which data collection is costly or dangerous. Enforcing constraint satisfaction is non-trivial, especially in offline settings, as there is a potential large discrepancy between the policy distribution and the data distribution, causing errors in estimating the value of safety constraints. We show that naïve approaches that combine techniques from safe RL and offline RL can only learn sub-optimal solutions. We thus develop a simple yet effective algorithm, Constraints Penalized Q-Learning (CPQ), to solve the problem. Our method admits the use of data generated by mixed behavior policies. We present a theoretical analysis and demonstrate empirically that our approach can learn robustly across a variety of benchmark control tasks, outperforming several baselines.
Robust Probabilistic Shielding for Safe Offline Reinforcement Learning
Radboud University Nijmegen The Netherlands；Ruhr University\,\&\,Radboud University Bochum Germany；Radboud University；Ruhr University\,\&\,Radboud University
Model-Based Proactive Cost Generation for Learning Safe Policies Offline with Limited Violation Data
National Key Laboratory for Novel Software Technology, Nanjing University（南京大学新型软件技术国家重点实验室）；School of Artificial Intelligence, Nanjing University（南京大学人工智能学院）；Polixir Technologies Nanjing, China（南京Polixir科技有限公司）；Tencent Game AI Center Shenzhen, China（腾讯深圳游戏AI中心）
Safe Evaluation For Offline Learning: Are We Ready To Deploy?
University of Alberta（阿尔伯塔大学）；University of Wisconsin – Madison（威斯康星大学麦迪逊分校）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Sony AI（索尼人工智能研究院）；Alberta Machine Intelligence Institute（阿尔伯塔机器智能研究院）
Towards Fast Safe Online Reinforcement Learning via Policy Finetuning
School of Electrical, Computer and Energy Engineering（电气、计算机与能源工程学院）；Arizona State University（亚利桑那州立大学）；School of Electrical Engineering and Computer Science（电气工程与计算机科学学院）；Washington State University（华盛顿州立大学）；Department of Computer Science（计算机科学系）；University of Houston（休斯顿大学）
Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Model
Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院）；School of Vehicle and Mobility, Tsinghua University（清华大学车辆与运载学院）；Department of Computer Science, The University of Hong Kong（香港大学计算机科学系）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
FOSP: Fine-tuning Offline Safe Policy through World Models
Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；College of Design and Engineering, National University of Singapore（新加坡国立大学设计与工程学院）
Offline Inverse Constrained Reinforcement Learning for Safe-Critical Decision Making in Healthcare
School of Computer Science and Technology（计算机科学与技术学院）；University of Science and Technology of China（中国科学技术大学）；School of Data Science（数据科学学院）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））
Individual Control Barrier Functions-Guided Diffusion Model for Safe Offline Multi-Agent Reinforcement Learning
Department of Electrical Engineering and Automation, Aalto University（阿尔托大学电气工程与自动化系）；School of Computing and Data Science, Xiamen University Malaysia（厦门大学马来西亚分校计算与数据科学学院）；Department of Computer Science, University of Toronto（多伦多大学计算机科学系）
Safe Deployment of Offline Reinforcement Learning via Input Convex Action Correction
Department of Chemical Engineering, Imperial College London（帝国理工学院化学工程系）；Shell Information Technology International BV（壳牌信息科技国际有限公司）；Shell Global Solutions International BV（壳牌全球解决方案国际有限公司）
CDCP: Conditional Diffusion Model with Contextual Prompts for Multi-task Offline Safe Reinforcement Learning
Tongji University（同济大学）；JD Explore Academy（京东探索研究院）；School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区网络科学与技术学院）；Berkeley AI Research Lab, University of California, Berkeley（加州大学伯克利分校伯克利人工智能研究实验室）；Technical University of Munich（慕尼黑工业大学）
