arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2207.06147cs.LGcs.AIstat.ML
收藏

CMDP中一种近优的原-对偶离策略学习方法

A Near-Optimal Primal-Dual Method for Off-Policy Learning in CMDP

Peking University(北京大学);National University of Singapore(新加坡国立大学);Beijing International Center for Mathematical Research(北京国际数学研究中心)

Fan Chen, Junyu Zhang, Zaiwen Wen

摘要:

作为安全强化学习的一个重要框架,约束马尔可夫决策过程(CMDP)在最近的文献中得到了广泛的研究。然而,尽管在各种同策略学习设置下有丰富的结果,仍然缺乏一些基本的理解离线CMDP问题,无论是算法设计和信息理论样本复杂度下限。在本文中,我们专注于解决CMDP问题,只有离线数据。通过引入单策略集中系数$C^*$的概念,建立了离线CMDP问题的$Ω\left(\frac{\min\left\{|\mathcal{S}||\mathcal{A}|,|\mathcal{S}|+I\right\} C^*}{(1-γ)^3ε^2}\right)$样本复杂度下界,其中$I$表示约束数目.通过引入一种简单但新颖的偏差控制机制,我们提出了一种接近最优的原始-对偶学习算法DPDL。该算法可证明地保证零约束违反,其样本复杂度匹配上述下限,除了一个$\tilde{\mathcal{O}}((1-γ)^{-1})$因子。还包括如何处理未知常数$C^*$和离线数据集上潜在的异步结构的全面讨论。

英文摘要:

As an important framework for safe Reinforcement Learning, the Constrained Markov Decision Process (CMDP) has been extensively studied in the recent literature. However, despite the rich results under various on-policy learning settings, there still lacks some essential understanding of the offline CMDP problems, in terms of both the algorithm design and the information theoretic sample complexity lower bound. In this paper, we focus on solving the CMDP problems where only offline data are available. By adopting the concept of the single-policy concentrability coefficient $C^*$, we establish an $Ω\left(\frac{\min\left\{|\mathcal{S}||\mathcal{A}|,|\mathcal{S}|+I\right\} C^*}{(1-γ)^3ε^2}\right)$ sample complexity lower bound for the offline CMDP problem, where $I$ stands for the number of constraints. By introducing a simple but novel deviation control mechanism, we propose a near-optimal primal-dual learning algorithm called DPDL. This algorithm provably guarantees zero constraint violation and its sample complexity matches the above lower bound except for an $\tilde{\mathcal{O}}((1-γ)^{-1})$ factor. Comprehensive discussion on how to deal with the unknown constant $C^*$ and the potential asynchronous structure on the offline dataset are also included.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑