arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2005.12900cs.LGcs.ITmath.ITmath.OCmath.STstat.MLstat.TH
收藏

在具备生成模型的基于模型的强化学习中突破样本量障碍

Breaking the Sample Size Barrier in Model-Based Reinforcement Learning with a Generative Model

University of Pennsylvania(宾夕法尼亚大学);Carnegie Mellon University(卡内基梅隆大学)

Gen Li, Yuting Wei, Yuejie Chi, Yuxin Chen

AI总结:

论文研究生成模型下强化学习的样本效率,证明扰动和保守的基于模型算法及普通模型规划可突破既有样本量障碍并达到极小极大最优。

摘要:

本文关注在可使用生成模型(或模拟器)假设下强化学习的样本效率。我们首先考虑具有状态空间$\mathcal{S}$和动作空间$\mathcal{A}$的$γ$折扣无限视野Markov决策过程(MDP)。尽管已有许多工作处理该问题,但样本复杂度与统计精度之间权衡的完整图景仍未确定。特别是,所有先前结果都受到严重样本量障碍的影响,即其所声称的统计保证仅在样本量至少超过$\frac{|\mathcal{S}||\mathcal{A}|}{(1-γ)^2}$时才成立。本文证明,一旦样本量超过$\frac{|\mathcal{S}||\mathcal{A}|}{1-γ}$的数量级(不计某个对数因子),两种算法——一种扰动的基于模型算法和一种保守的基于模型算法——即具有极小极大最优性,从而克服了该障碍。除无限视野MDP之外,我们进一步研究时间非齐次有限视野MDP,并证明普通的基于模型规划算法就足以在任意给定目标精度水平下达到极小极大最优的样本复杂度。据我们所知,本工作首次给出了可容纳整个样本量范围的极小极大最优保证;超过该范围后,寻找有意义策略在信息论上是不可行的。

英文摘要:

This paper is concerned with the sample efficiency of reinforcement learning, assuming access to a generative model (or simulator). We first consider $γ$-discounted infinite-horizon Markov decision processes (MDPs) with state space $\mathcal{S}$ and action space $\mathcal{A}$. Despite a number of prior works tackling this problem, a complete picture of the trade-offs between sample complexity and statistical accuracy is yet to be determined. In particular, all prior results suffer from a severe sample size barrier, in the sense that their claimed statistical guarantees hold only when the sample size exceeds at least $\frac{|\mathcal{S}||\mathcal{A}|}{(1-γ)^2}$. The current paper overcomes this barrier by certifying the minimax optimality of two algorithms -- a perturbed model-based algorithm and a conservative model-based algorithm -- as soon as the sample size exceeds the order of $\frac{|\mathcal{S}||\mathcal{A}|}{1-γ}$ (modulo some log factor). Moving beyond infinite-horizon MDPs, we further study time-inhomogeneous finite-horizon MDPs, and prove that a plain model-based planning algorithm suffices to achieve minimax-optimal sample complexity given any target accuracy level. To the best of our knowledge, this work delivers the first minimax-optimal guarantees that accommodate the entire range of sample sizes (beyond which finding a meaningful policy is information theoretically infeasible).

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑