面向 $q_π$-可实现约束 MDP 中局部规划的置信自然策略梯度
Confident Natural Policy Gradient for Local Planning in $q_π$-realizable Constrained MDPs
AI总结：
针对无限状态 $q_π$-可实现 CMDP，提出基于局部访问与离策略评估的置信自然策略梯度原对偶算法，实现多项式样本复杂度并高概率获得近优且严格满足约束的策略。
摘要：
约束 Markov 决策过程（CMDP）框架是一种重要的强化学习方法，用于在最大化累积奖励的同时施加安全性或其他关键目标。然而，目前对于如何在可能具有无限状态数的 CMDP 环境中高效学习仍在研究之中，尤其是在对价值函数应用函数逼近时。本文研究在 $q_π$-可实现性下采用线性函数逼近的学习问题，其中所有策略的价值函数都可通过已知特征映射线性表示；已知这一设置比其他线性设置更一般且更具挑战性。利用局部访问模型，我们提出一种新的原对偶算法，该算法在经过 $\ ilde{O}(\ ext{poly}(d) ε^{-3})$ 次查询后，以高概率输出一个严格满足约束、同时相对于奖励函数近乎最优地优化价值的策略。这里，$d$ 是特征维度，$ε>0$ 是给定误差。该算法依赖精心设计的离策略评估过程，利用历史数据评估策略，进而通过策略梯度为策略更新提供信息并节省样本。据我们所知，这是在 $q_π$-可实现设置下首次为 CMDP 取得多项式样本复杂度的结果。
英文摘要：
The constrained Markov decision process (CMDP) framework emerges as an important reinforcement learning approach for imposing safety or other critical objectives while maximizing cumulative reward. However, the current understanding of how to learn efficiently in a CMDP environment with a potentially infinite number of states remains under investigation, particularly when function approximation is applied to the value functions. In this paper, we address the learning problem given linear function approximation with $q_π$-realizability, where the value functions of all policies are linearly representable with a known feature map, a setting known to be more general and challenging than other linear settings. Utilizing a local-access model, we propose a novel primal-dual algorithm that, after $\tilde{O}(\text{poly}(d) ε^{-3})$ queries, outputs with high probability a policy that strictly satisfies the constraints while nearly optimizing the value with respect to a reward function. Here, $d$ is the feature dimension and $ε> 0$ is a given error. The algorithm relies on a carefully crafted off-policy evaluation procedure to evaluate the policy using historical data, which informs policy updates through policy gradients and conserves samples. To our knowledge, this is the first result achieving polynomial sample complexity for CMDP in the $q_π$-realizable setting.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
