arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2405.16601cs.LG
收藏

A CMDP-within-online框架用于元安全强化学习

A CMDP-within-online framework for Meta-Safe Reinforcement Learning

Virginia Tech(弗吉尼亚理工大学);UC Berkeley(加州大学伯克利分校)

Vanshaj Khattar, Yuhao Ding, Bilgehan Sel, Javad Lavaei, Ming Jin

AI总结:

本文提出CMDP-within-online框架解决元安全强化学习问题,通过梯度元学习获得任务平均遗憾界,并设计不精确在线学习元算法,实现约束满足与奖励优化,实验验证其有效性。

摘要:

元强化学习已被广泛用作一种学习如何学习的框架,以有限的经验解决未见过的任务。然而,现有工作尚未充分解决约束违反的问题,这限制了它们在实际场景中的应用。在本文中,我们通过CMDP-within-online框架研究元安全强化学习(Meta-SRL)问题,在这一重要场景中建立了首个可证明的保证。我们利用基于梯度的元学习,获得了奖励最大化(最优性差距)和约束违反的任务平均遗憾界,并表明在静态环境中任务平均最优性差距和约束满足度随任务相似性而改善,在动态环境中则随任务相关性而改善。在使该框架实用化时出现了若干技术挑战。为此,我们提出了一种元算法,该算法对由离策略平稳分布校正估计的每个任务内最优性差距和约束违反的上界执行不精确的在线学习。此外,我们使学习率能够针对每个任务进行自适应调整,并将我们的方法扩展到存在动态变化的竞争性预言机(oracle)的设置。最后,我们进行了实验以证明我们方法的有效性。

英文摘要:

Meta-reinforcement learning has widely been used as a learning-to-learn framework to solve unseen tasks with limited experience. However, the aspect of constraint violations has not been adequately addressed in the existing works, making their application restricted in real-world settings. In this paper, we study the problem of meta-safe reinforcement learning (Meta-SRL) through the CMDP-within-online framework to establish the first provable guarantees in this important setting. We obtain task-averaged regret bounds for the reward maximization (optimality gap) and constraint violations using gradient-based meta-learning and show that the task-averaged optimality gap and constraint satisfaction improve with task-similarity in a static environment or task-relatedness in a dynamic environment. Several technical challenges arise when making this framework practical. To this end, we propose a meta-algorithm that performs inexact online learning on the upper bounds of within-task optimality gap and constraint violations estimated by off-policy stationary distribution corrections. Furthermore, we enable the learning rates to be adapted for every task and extend our approach to settings with a competing dynamically changing oracle. Finally, experiments are conducted to demonstrate the effectiveness of our approach.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑