从奖励未知的演示中学习安全约束
Learning Safety Constraints from Demonstrations with Unknown Rewards
AI总结：
提出CoCoRL，在奖励和环境动态未知时从多奖励安全演示中推断CMDP共享约束，并保证安全、可收敛及跨环境迁移。
摘要：
我们提出了面向强化学习的凸约束学习（Convex Constraint Learning for Reinforcement Learning，CoCoRL），这是一种从一组可能具有不同奖励函数的安全演示中推断约束马尔可夫决策过程（Constrained Markov Decision Process，CMDP）中共享约束的新方法。以往工作局限于奖励已知或环境动态完全已知的演示，而CoCoRL能够在不了解环境动态的情况下，从具有不同未知奖励的演示中学习约束。CoCoRL基于演示构建一个凸安全集，即使对于可能次优但安全的演示，也可证明地保证安全性。对于近似最优演示，CoCoRL会收敛到真实安全集，且没有策略遗憾。我们在网格世界环境和具有多个约束的驾驶模拟中评估了CoCoRL。CoCoRL学习到的约束能够带来安全的驾驶行为。重要的是，我们可以将学习到的约束安全迁移到不同任务和环境。相比之下，基于逆强化学习（Inverse Reinforcement Learning，IRL）的替代方法通常表现较差，并会学习到不安全的策略。
英文摘要：
We propose Convex Constraint Learning for Reinforcement Learning (CoCoRL), a novel approach for inferring shared constraints in a Constrained Markov Decision Process (CMDP) from a set of safe demonstrations with possibly different reward functions. While previous work is limited to demonstrations with known rewards or fully known environment dynamics, CoCoRL can learn constraints from demonstrations with different unknown rewards without knowledge of the environment dynamics. CoCoRL constructs a convex safe set based on demonstrations, which provably guarantees safety even for potentially sub-optimal (but safe) demonstrations. For near-optimal demonstrations, CoCoRL converges to the true safe set with no policy regret. We evaluate CoCoRL in gridworld environments and a driving simulation with multiple constraints. CoCoRL learns constraints that lead to safe driving behavior. Importantly, we can safely transfer the learned constraints to different tasks and environments. In contrast, alternative methods based on Inverse Reinforcement Learning (IRL) often exhibit poor performance and learn unsafe policies.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
