Offline Preference-based Reinforcement Learning (PbRL) learns rewards and policies aligned with human preferences without the need for extensive reward engineering and direct interaction with human annotators. However, ensuring safety remains a critical challenge across many domains and tasks. Previous works on safe RL from human feedback (RLHF) first learn reward and cost models from offline data, then use constrained RL to optimize a safe policy. While such an approach works in the contextual bandits settings (LLMs), in long horizon continuous control tasks, errors in rewards and costs accumulate, leading to impairment in performance when used with constrained RL methods. To address these challenges, (a) instead of indirectly learning policies (from rewards and costs), we introduce a framework that learns a policy directly based on pairwise preferences regarding the agent's behavior in terms of rewards, as well as binary labels indicating the safety of trajectory segments; (b) we propose \textsc{PreSa} (Preference and Safety Alignment), a method that combines preference learning module with safety alignment in a constrained optimization problem. This optimization problem is solved within a Lagrangian paradigm that directly learns reward-maximizing safe policy \textit{without explicitly learning reward and cost models}, avoiding the need for constrained RL; (c) we evaluate our approach on continuous control tasks with both synthetic and real human feedback. Empirically, our method successfully learns safe policies with high rewards, outperforming state-of-the-art baselines, and offline safe RL approaches with ground-truth reward and cost.
Robust Probabilistic Shielding for Safe Offline Reinforcement Learning
Radboud University Nijmegen The Netherlands；Ruhr University\,\&\,Radboud University Bochum Germany；Radboud University；Ruhr University\,\&\,Radboud University
Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Model
Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院）；School of Vehicle and Mobility, Tsinghua University（清华大学车辆与运载学院）；Department of Computer Science, The University of Hong Kong（香港大学计算机科学系）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
Safe Evaluation For Offline Learning: Are We Ready To Deploy?
University of Alberta（阿尔伯塔大学）；University of Wisconsin – Madison（威斯康星大学麦迪逊分校）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Sony AI（索尼人工智能研究院）；Alberta Machine Intelligence Institute（阿尔伯塔机器智能研究院）
Towards Fast Safe Online Reinforcement Learning via Policy Finetuning
School of Electrical, Computer and Energy Engineering（电气、计算机与能源工程学院）；Arizona State University（亚利桑那州立大学）；School of Electrical Engineering and Computer Science（电气工程与计算机科学学院）；Washington State University（华盛顿州立大学）；Department of Computer Science（计算机科学系）；University of Houston（休斯顿大学）
Safe Deployment of Offline Reinforcement Learning via Input Convex Action Correction
Department of Chemical Engineering, Imperial College London（帝国理工学院化学工程系）；Shell Information Technology International BV（壳牌信息科技国际有限公司）；Shell Global Solutions International BV（壳牌全球解决方案国际有限公司）
Offline Inverse Constrained Reinforcement Learning for Safe-Critical Decision Making in Healthcare
School of Computer Science and Technology（计算机科学与技术学院）；University of Science and Technology of China（中国科学技术大学）；School of Data Science（数据科学学院）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））
Variational OOD State Correction for Offline Reinforcement Learning
College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, MIIT Key Laboratory of Pattern Analysis and Machine Intelligence（计算机科学与技术学院，南京航空航天大学，信息科技部模式分析与机器智能重点实验室）
Robo-ValueRL: Reliable Value Estimation for Offline-to-Online Reinforcement Learning
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；Beijing Innovation Center of Humanoid Robotics（北京人形机器人创新中心）；Beijing Forestry University（北京林业大学）；Peking University（北京大学）；Microsoft Research（微软研究院）
Data Center Cooling System Optimization Using Offline Reinforcement Learning
Institute for AI Industry Research, Tsinghua University（清华大学人工智能产业研究院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Global Data Solutions Co., Ltd.（全球数据解决方案有限公司）
Model-Based Proactive Cost Generation for Learning Safe Policies Offline with Limited Violation Data
National Key Laboratory for Novel Software Technology, Nanjing University（南京大学新型软件技术国家重点实验室）；School of Artificial Intelligence, Nanjing University（南京大学人工智能学院）；Polixir Technologies Nanjing, China（南京Polixir科技有限公司）；Tencent Game AI Center Shenzhen, China（腾讯深圳游戏AI中心）
