Though deep reinforcement learning (DRL) has obtained substantial success, it may encounter catastrophic failures due to the intrinsic uncertainty of both transition and observation. Most of the existing methods for safe reinforcement learning can only handle transition disturbance or observation disturbance since these two kinds of disturbance affect different parts of the agent; besides, the popular worst-case return may lead to overly pessimistic policies. To address these issues, we first theoretically prove that the performance degradation under transition disturbance and observation disturbance depends on a novel metric of Value Function Range (VFR), which corresponds to the gap in the value function between the best state and the worst state. Based on the analysis, we adopt conditional value-at-risk (CVaR) as an assessment of risk and propose a novel reinforcement learning algorithm of CVaR-Proximal-Policy-Optimization (CPPO) which formalizes the risk-sensitive constrained optimization problem by keeping its CVaR under a given threshold. Experimental results show that CPPO achieves a higher cumulative reward and is more robust against both observation and transition disturbances on a series of continuous control tasks in MuJoCo.
CDCP: Conditional Diffusion Model with Contextual Prompts for Multi-task Offline Safe Reinforcement Learning
Tongji University（同济大学）；JD Explore Academy（京东探索研究院）；School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区网络科学与技术学院）；Berkeley AI Research Lab, University of California, Berkeley（加州大学伯克利分校伯克利人工智能研究实验室）；Technical University of Munich（慕尼黑工业大学）
Long and Short-Term Constraints Driven Safe Reinforcement Learning for Autonomous Driving
School of Artificial Intelligence, Hubei University（湖北大学人工智能学院）；Worcester Polytechnic Institute（伍斯特理工学院）；Chinese Academy of Sciences（中国科学院）；WAYTOUS Inc.（万途科技公司）；Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)（广东省人工智能与数字经济实验室（深圳））
Model-based Safe Deep Reinforcement Learning via a Constrained Proximal Policy Optimization Algorithm
Indian Institute of Science（印度科学学院）
在大多数强化学习（RL）算法的初始训练迭代期间，代理执行大量随机探索步骤。在现实世界中，这可能会限制这些算法的实用性，因为它可能导致潜在的危险行为。因此，安全探索是在现实世界中应用RL算法的一个关键问题。这个问题最近在约束马尔可夫决策过程（CMDP）框架下得到了很好的研究，其中除了单阶段奖励外，代理还根据状态转换接收单阶段成本或惩罚。规定的成本函数负责将任何给定时间步的不良行为映射到标量值。然后，目标是找到一个可行的政策，最大限度地提高回报，同时限制成本回报低于规定的阈值，在训练和部署。我们提出了一种基于策略模型的安全深度RL算法，在该算法中，我们以在线方式学习环境的过渡动态，并使用基于拉格朗日松弛的近端策略优化找到可行的最优策略。我们使用不同初始化的神经网络的集成来解决环境模型学习过程中所面临的认知和任意不确定性问题。我们使用具有挑战性的安全强化学习基准-Open AI Safety Gym，将我们的方法与约束RL中的相关无模型和基于模型的方法进行比较。我们证明，我们的算法是更有效的样本，并在较低的累积危险违规的结果相比，受约束的无模型的方法。此外，我们的方法显示出更好的回报性能比其他约束的基于模型的方法在文献中。
RADE: Learning Risk-Adjustable Driving Environment via Multi-Agent Conditional Diffusion
Department of Civil and Environmental Engineering, University of Michigan（密歇根大学土木与环境工程系）；Department of Computer Science, University of Hong Kong（香港大学计算机科学系）；University of Michigan Transportation Research Institute（密歇根大学交通研究研究院）
