我们认为，在最佳的价值函数位于接近的跨度的特征地图的假设下，在固定水平的MDP与生成模型的局部规划。生成模型提供了对MDP的本地访问：规划器可以要求从先前返回的状态和任意动作进行随机转换，并且仅可访问此过程中遇到的状态的特征。与之前的工作（例如Latestival et al.（2020））相反，假设所有策略都是线性可实现的，我们考虑了单个线性可实现（确定性）策略的显著宽松假设。Weisz等人（2020）最近的一个下限确定了当最优策略的动作价值函数是线性可实现的时，相关问题需要指数数量的查询，无论是在$H$（MDP的视野）还是$d$（特征映射的维度）中。它们的构建关键依赖于具有指数级大的动作集。相比之下，在这项工作中，我们建立了多$(H,d)$规划是可能的状态价值函数实现时，动作集有一个恒定的大小。特别是，我们提出了TensorPlan算法，它使用聚$((dH/δ)^A)$模拟器查询，以找到一个$δ$最优的政策相对于任何确定性的政策，其中的价值函数是线性实现与一些有界参数。这是第一个只使用单个竞争价值函数的线性可实现性来给出多项式查询复杂度保证的算法。计算成本是否同样有界仍然是一个悬而未决的问题。我们将上界扩展到近可实现的情况下，无限地平线贴现设置。我们还提出了一个下界在无限地平线的情节设置：Plancies，实现恒定的次优需要指数许多查询，无论是在$d$或行动的数量。
英文摘要：
We consider local planning in fixed-horizon MDPs with a generative model under the assumption that the optimal value function lies close to the span of a feature map. The generative model provides a local access to the MDP: The planner can ask for random transitions from previously returned states and arbitrary actions, and features are only accessible for states that are encountered in this process. As opposed to previous work (e.g. Lattimore et al. (2020)) where linear realizability of all policies was assumed, we consider the significantly relaxed assumption of a single linearly realizable (deterministic) policy. A recent lower bound by Weisz et al. (2020) established that the related problem when the action-value function of the optimal policy is linearly realizable requires an exponential number of queries, either in $H$ (the horizon of the MDP) or $d$ (the dimension of the feature mapping). Their construction crucially relies on having an exponentially large action set. In contrast, in this work, we establish that poly$(H,d)$ planning is possible with state value function realizability whenever the action set has a constant size. In particular, we present the TensorPlan algorithm which uses poly$((dH/δ)^A)$ simulator queries to find a $δ$-optimal policy relative to any deterministic policy for which the value function is linearly realizable with some bounded parameter. This is the first algorithm to give a polynomial query complexity guarantee using only linear-realizability of a single competing value function. Whether the computation cost is similarly bounded remains an open question. We extend the upper bound to the near-realizable case and to the infinite-horizon discounted setup. We also present a lower bound in the infinite-horizon episodic setting: Planners that achieve constant suboptimality need exponentially many queries, either in $d$ or the number of actions.
Frozen Policy Iteration: Computationally Efficient RL under Linear $Q^π$ Realizability for Deterministic Dynamics
School of EECS, Peking University（北京大学电子工程系）；Department of Computer Science and Engineering, HKUST（香港科技大学计算机科学与工程系）；CFCS and School of Computer Science, Peking University（北京大学计算机学院）
Convex-Analytic View of Query Selection for Safety Verification of Markov Decision Processes
The Institute of Statistical Mathematics（统计数学研究所）；Tohoku University（东北大学）；RIKEN AIP（理化学研究所人工智能中心）；CyberAgent；National Institute of Informatics（信息学国立研究所）
Adaptive Resolving Methods for Markov Decision Processes with Function Approximations
Department of Industrial Engineering & Decision Analytics, The Hong Kong University of Science and Technology（香港科学与技术大学工业工程与决策分析系）；Department of Management Science & Engineering, Stanford University（斯坦福大学管理科学与工程系）
