Meta-reinforcement learning has widely been used as a learning-to-learn framework to solve unseen tasks with limited experience. However, the aspect of constraint violations has not been adequately addressed in the existing works, making their application restricted in real-world settings. In this paper, we study the problem of meta-safe reinforcement learning (Meta-SRL) through the CMDP-within-online framework to establish the first provable guarantees in this important setting. We obtain task-averaged regret bounds for the reward maximization (optimality gap) and constraint violations using gradient-based meta-learning and show that the task-averaged optimality gap and constraint satisfaction improve with task-similarity in a static environment or task-relatedness in a dynamic environment. Several technical challenges arise when making this framework practical. To this end, we propose a meta-algorithm that performs inexact online learning on the upper bounds of within-task optimality gap and constraint violations estimated by off-policy stationary distribution corrections. Furthermore, we enable the learning rates to be adapted for every task and extend our approach to settings with a competing dynamically changing oracle. Finally, experiments are conducted to demonstrate the effectiveness of our approach.
Near-Constant Strong Violation and Last-Iterate Convergence for Online CMDPs via Decaying Safety Margins
School of Informatics, University of Edinburgh, Edinburgh, United Kingdom(爱丁堡大学信息学院);School of Mathematics, University of Edinburgh, Edinburgh, United Kingdom(爱丁堡大学数学学院)
Provably Efficient RL under Episode-Wise Safety in Constrained MDPs with Linear Function Approximation
The University of Tokyo(东京大学);New Jersey Institute of Technology(新泽西理工学院);OMRON SINIC X(欧姆龙SINIC X);NexaScience;Kyoto University(京都大学);Institute of Science Tokyo(东京科学大学);DENSO IT Laboratory(电装IT实验室)
SCPO: Safe Reinforcement Learning with Safety Critic Policy Optimization
Technical University of Munich(慕尼黑工业大学);Peking University(北京大学);BIGAI(北京通用人工智能研究院);King's College London(伦敦国王学院);Tongji University(同济大学);University College London(伦敦大学学院)
Certifiable Safe RLHF: Semantic Grounding and Fixed Penalty Constraint Optimization for Safer LLM Alignment
Department of Electrical and Computer Engineering(电气与计算机工程系);New Jersey Institute of Technology(新泽西理工学院);Department of Computer Engineering(计算机工程系);Heritage Institute of Technology(遗产理工学院)
Near-Optimal Primal-Dual Algorithm for Learning Linear Mixture CMDPs with Adversarial Rewards
Department of Industrial and Systems Engineering, KAIST(韩国科学技术院工业与系统工程系);Department of Mathematical Sciences, Seoul National University(首尔大学数学科学系);Research Institute of Mathematics, Seoul National University(首尔大学数学研究所);Interdisciplinary Program in Artificial Intelligence, Seoul National University(首尔大学人工智能跨学科项目);Korea Institute for Advanced Study(韩国高等研究院)
Enforcing Hard Constraints with Soft Barriers: Safe Reinforcement Learning in Unknown Stochastic Environments
Northwestern University(西北大学);University of California, Berkeley(加州大学伯克利分校);University of Pennsylvania(宾夕法尼亚大学);Yale University(耶鲁大学);University of Liverpool(利物浦大学)
Bi-Level Online Provisioning and Scheduling with Switching Costs and Cross-Level Constraints
Dept. of Electrical Engineering University at Buffalo, Buffalo, NY(电气工程系 网络布法罗大学);Computer Engineering The Ohio State University, Columbus, OH(计算机工程 俄亥俄州立大学)
Direct Behavior Specification via Constrained Reinforcement Learning
Institut d’intelligence aritficielle du Québec (Mila)(魁北克人工智能研究所(Mila));École Polytechnique de Montréal(蒙特利尔理工学院);Ubisoft La Forge(育碧La Forge实验室);Université de Montréal(蒙特利尔大学);ServiceNow(ServiceNow公司)
SafeOR-Gym: A Benchmark Suite for Safe Reinforcement Learning Algorithms on Practical Operations Research Problems
Davidson School of Chemical Engineering, Purdue University(普渡大学化学工程学院);Energy Sciences, ExxonMobil Technology and Engineering Company(埃克森美孚技术与工程公司能源科学部)
Achieving Instance-dependent Sample Complexity for Constrained Markov Decision Process
Department of Industrial Engineering & Decision Analytics, Hong Kong University of Science and Technology(香港科技大学工业工程与决策分析系);Institute for Computational and Mathematical Engineering, Stanford University(斯坦福大学计算与数学工程研究所);Department of Management Science & Engineering, Stanford University(斯坦福大学管理科学与工程系)
Safe Multi-Agent Deep Reinforcement Learning for Privacy-Aware Edge-Device Collaborative DNN Inference
School of Future Science and Engineering, Soochow University(未来科学与工程学院,苏州大学);College of Computer and Information Sciences, Fujian Agriculture and Forestry University(计算机与信息科学学院,福建农林大学);School of Information and Control Engineering, China University of Mining and Technology(信息与控制工程学院,中国矿业大学);Department of Control Science and Engineering, Tongji University(控制科学与工程学院,同济大学);School of Computing Technologies, RMIT University(计算技术学院,拉筹伯大学)
Safe and Sustainable Electric Bus Charging Scheduling with Constrained Hierarchical DRL
College of Engineering, University of Guelph(圭尔夫大学工程学院);EthicalAI;College of Computing and Data Science, Nanyang Technological University(南洋理工大学计算与数据科学学院)
Model-based Safe Deep Reinforcement Learning via a Constrained Proximal Policy Optimization Algorithm
Indian Institute of Science(印度科学学院)
在大多数强化学习(RL)算法的初始训练迭代期间,代理执行大量随机探索步骤。在现实世界中,这可能会限制这些算法的实用性,因为它可能导致潜在的危险行为。因此,安全探索是在现实世界中应用RL算法的一个关键问题。这个问题最近在约束马尔可夫决策过程(CMDP)框架下得到了很好的研究,其中除了单阶段奖励外,代理还根据状态转换接收单阶段成本或惩罚。规定的成本函数负责将任何给定时间步的不良行为映射到标量值。然后,目标是找到一个可行的政策,最大限度地提高回报,同时限制成本回报低于规定的阈值,在训练和部署。我们提出了一种基于策略模型的安全深度RL算法,在该算法中,我们以在线方式学习环境的过渡动态,并使用基于拉格朗日松弛的近端策略优化找到可行的最优策略。我们使用不同初始化的神经网络的集成来解决环境模型学习过程中所面临的认知和任意不确定性问题。我们使用具有挑战性的安全强化学习基准-Open AI Safety Gym,将我们的方法与约束RL中的相关无模型和基于模型的方法进行比较。我们证明,我们的算法是更有效的样本,并在较低的累积危险违规的结果相比,受约束的无模型的方法。此外,我们的方法显示出更好的回报性能比其他约束的基于模型的方法在文献中。
Embedding Safety into RL: A New Take on Trust Region Methods
Max Planck Institute for Human Cognitive and Brain Sciences(马克斯·普朗克人类认知与脑科学研究所);Center for Scalable Data Analytics and Artificial Intelligence(可扩展数据分析与人工智能中心)