arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2304.03081cs.LGcs.AI
收藏

通过学习不良轨迹的时间模式进行安全的MDP规划并避免负面副作用

Safe MDP Planning by Learning Temporal Patterns of Undesirable Trajectories and Averting Negative Side Effects

Singapore Management University(新加坡管理大学);University of Toronto(多伦多大学)

Siow Meng Low, Akshat Kumar, Scott Sanner

AI总结:

针对状态表示保真度不足导致的负面副作用(NSEs)问题,提出通过监督学习从轨迹分类标签中学习非马尔可夫安全模式,并利用Lagrange乘子法将安全模型与MDP整合于单一计算图中,在优化总回报的同时有效满足复杂安全约束。

摘要:

在安全的 MDP 规划中,通常使用基于当前状态和动作的成本函数来规定安全方面。在现实世界中,所使用的状态表示往往缺乏足够的保真度来规定此类安全约束。基于不完整的模型进行操作通常会产生意外的负面副作用(NSEs)。为了应对这些挑战,首先,我们将安全信号与状态-动作轨迹(而不仅仅是即时的状态-动作)相关联。这使得我们的安全模型具有高度的通用性。我们还假设为不同的轨迹提供了分类安全标签,而不是数值成本函数,后者更难由问题设计者规定。然后,我们采用监督学习模型来学习此类非马尔可夫安全模式。其次,我们开发了一种 Lagrange 乘子方法,该方法将安全模型和底层 MDP 模型整合在单个计算图中,以促进智能体学习安全行为。最后,我们在各种离散和连续领域上的实证结果表明,这种方法能够满足复杂的非马尔可夫安全约束,同时优化智能体的总回报,具有高度的可扩展性,并且也优于先前用于马尔可夫 NSEs 的最佳方法。

英文摘要:

In safe MDP planning, a cost function based on the current state and action is often used to specify safety aspects. In the real world, often the state representation used may lack sufficient fidelity to specify such safety constraints. Operating based on an incomplete model can often produce unintended negative side effects (NSEs). To address these challenges, first, we associate safety signals with state-action trajectories (rather than just an immediate state-action). This makes our safety model highly general. We also assume categorical safety labels are given for different trajectories, rather than a numerical cost function, which is harder to specify by the problem designer. We then employ a supervised learning model to learn such non-Markovian safety patterns. Second, we develop a Lagrange multiplier method, which incorporates the safety model and the underlying MDP model in a single computation graph to facilitate agent learning of safe behaviors. Finally, our empirical results on a variety of discrete and continuous domains show that this approach can satisfy complex non-Markovian safety constraints while optimizing an agent's total returns, is highly scalable, and is also better than the previous best approach for Markovian NSEs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑