Designing a safe policy for uncertain environments is crucial in real-world control systems. However, this challenge remains inadequately addressed within the Markov decision process (MDP) framework. This paper presents the first algorithm guaranteed to identify a near-optimal policy in a robust constrained MDP (RCMDP), where an optimal policy minimizes cumulative cost while satisfying constraints in the worst-case scenario across a set of environments. We first prove that the conventional policy gradient approach to the Lagrangian max-min formulation can become trapped in suboptimal solutions. This occurs when its inner minimization encounters a sum of conflicting gradients from the objective and constraint functions. To address this, we leverage the epigraph form of the RCMDP problem, which resolves the conflict by selecting a single gradient from either the objective or the constraints. Building on the epigraph form, we propose a bisection search algorithm with a policy gradient subroutine and prove that it identifies an $\varepsilon$-optimal policy in an RCMDP with $\tilde{\mathcal{O}}(\varepsilon^{-4})$ robust policy evaluations.
Offline Reinforcement Learning with Instrumental Variables in Confounded Markov Decision Processes
Northwestern University(西北大学);George Washington University(乔治华盛顿大学);Yale University(耶鲁大学);Johns Hopkins University(约翰斯·霍普金斯大学);University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校)
Quantum open system identification via global optimization: Optimally accurate Markovian models of open systems from time-series data
Tulane University(杜兰大学);DEVCOM Army Research Laboratory(DEVCOM陆军研究实验室);University of Massachusetts at Boston(马萨诸塞大学波士顿分校);HSBC Lab(汇丰实验室);Czech Technical University in Prague(捷克布拉格技术大学);Athena Research and Innovation Center(雅典娜研究与创新中心)
Non-asymptotic Identification of LTI Systems from a Single Trajectory
Department of Electrical and Computer Engineering, University of California, Riverside, CA(加州大学河滨分校电子工程与计算机科学系);Electrical Engineering and Computer Science Department, University of Michigan, Ann Arbor, MI(密歇根大学安娜堡分校电气工程与计算机科学系)