arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.16286cs.LGmath.OC
收藏

通过epigraph形式在鲁棒约束马尔可夫决策过程中的近优策略识别

Near-Optimal Policy Identification in Robust Constrained Markov Decision Processes via Epigraph Form

The University of Tokyo(东京大学);OMRON SINIC X Corporation(OMRON SINIC X公司);Kyoto University(京都大学);Osaka University(大阪大学)

Toshinori Kitamura, Tadashi Kozuno, Wataru Kumagai, Kenta Hoshino, Yohei Hosoe, Kazumi Kasaura, Masashi Hamaya, Paavo Parmas, Yutaka Matsuo

AI总结:

本文提出一种算法,通过epigraph形式解决鲁棒约束马尔可夫决策过程中的近优策略识别问题,证明该算法在$\tilde{\mathcal{O}}(\varepsilon^{-4})$次鲁棒策略评估内找到$\varepsilon$-最优策略。

摘要:

在不确定环境中设计安全策略对现实世界控制系统至关重要。然而,这一挑战在马尔可夫决策过程(MDP)框架内仍未得到充分解决。本文提出了第一个能够识别鲁棒约束马尔可夫决策过程(RCMDP)中近优策略的算法,其中最优策略在一组环境中最小化累积成本同时满足约束的最坏情况。我们首先证明传统策略梯度方法在处理拉格朗日max-min公式时可能陷入次优解,当其内部最小化遇到目标函数和约束函数的冲突梯度之和时。为了解决这一问题,我们利用RCMDP问题的epigraph形式,通过选择目标或约束中的单一梯度来解决冲突。基于epigraph形式,我们提出了一种带有策略梯度子程序的二分搜索算法,并证明其在RCMDP中找到$\varepsilon$-最优策略,所需鲁棒策略评估为$\tilde{\mathcal{O}}(\varepsilon^{-4})$。

英文摘要:

Designing a safe policy for uncertain environments is crucial in real-world control systems. However, this challenge remains inadequately addressed within the Markov decision process (MDP) framework. This paper presents the first algorithm guaranteed to identify a near-optimal policy in a robust constrained MDP (RCMDP), where an optimal policy minimizes cumulative cost while satisfying constraints in the worst-case scenario across a set of environments. We first prove that the conventional policy gradient approach to the Lagrangian max-min formulation can become trapped in suboptimal solutions. This occurs when its inner minimization encounters a sum of conflicting gradients from the objective and constraint functions. To address this, we leverage the epigraph form of the RCMDP problem, which resolves the conflict by selecting a single gradient from either the objective or the constraints. Building on the epigraph form, we propose a bisection search algorithm with a policy gradient subroutine and prove that it identifies an $\varepsilon$-optimal policy in an RCMDP with $\tilde{\mathcal{O}}(\varepsilon^{-4})$ robust policy evaluations.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑