arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2209.15634cs.LGcs.AIstat.ML
收藏

强化学习中样本有效函数逼近的一般框架

A General Framework for Sample-Efficient Function Approximation in Reinforcement Learning

University of California, Los Angeles(加州大学洛杉矶分校);University of California, Berkeley(加州大学伯克利分校)

Zixiang Chen, Chris Junchi Li, Angela Yuan, Quanquan Gu, Michael I. Jordan

摘要:

随着对处理大型状态空间和动作空间的需求不断增加,广义函数逼近已成为强化学习(RL)中的一项关键技术。在本文中,我们提出了一个统一基于模型和无模型的RL的一般框架,以及一个可接受的贝尔曼特征(ABC)类,它包含了几乎所有的马尔可夫决策过程(MDP)模型在文献中的易处理的RL。我们提出了一种新的估计函数与可分解的结构特性,优化为基础的探索和功能的逃避维度作为ABC类的复杂性度量。在我们的框架下,提出了一种新的样本有效的算法,即基于优化的近似探索(OPERA),实现遗憾界匹配或改善各种MDP模型的最佳知名的结果。特别是,对于具有低Witness秩的MDP,在稍微更强的假设下,OPERA将最先进的样本复杂度结果提高了$dH$的因子。我们的框架提供了一个通用的接口来设计和分析新的RL模型和算法。

英文摘要:

With the increasing need for handling large state and action spaces, general function approximation has become a key technique in reinforcement learning (RL). In this paper, we propose a general framework that unifies model-based and model-free RL, and an Admissible Bellman Characterization (ABC) class that subsumes nearly all Markov Decision Process (MDP) models in the literature for tractable RL. We propose a novel estimation function with decomposable structural properties for optimization-based exploration and the functional eluder dimension as a complexity measure of the ABC class. Under our framework, a new sample-efficient algorithm namely OPtimization-based ExploRation with Approximation (OPERA) is proposed, achieving regret bounds that match or improve over the best-known results for a variety of MDP models. In particular, for MDPs with low Witness rank, under a slightly stronger assumption, OPERA improves the state-of-the-art sample complexity results by a factor of $dH$. Our framework provides a generic interface to design and analyze new RL models and algorithms.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑