arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2110.06394cs.LGcs.AImath.OCstat.ML
收藏

基于线性函数逼近的无报酬基于模型的强化学习

Reward-Free Model-Based Reinforcement Learning with Linear Function Approximation

University of California, Los Angeles(加州大学洛杉矶分校)

Weitong Zhang, Dongruo Zhou, Quanquan Gu

摘要:

研究了基于模型的无报酬线性函数逼近的强化学习在情景马尔可夫决策过程中的应用。在此设置中,代理分两个阶段工作。在探索阶段,智能体与环境进行交互,并在没有奖励的情况下收集样本。在规划阶段,智能体被赋予特定的奖励函数,并使用从探索阶段收集的样本来学习一个好的策略。我们提出了一个新的可证明有效的算法,称为UCRL RFE下的线性混合MDP假设,其中的转移概率内核的MDP可以通过一个线性函数参数化的状态,动作,和下一个状态的三元组上定义的某些特征映射。我们发现,要获得$ε$ -最优的政策,任意奖励函数,UCRL-RFE需要在最多$\tilde{\mathcal{O}}(H^5d^2ε^{-2})$情节在探索阶段的样本。这里,$H$是片段的长度,$d$是特征映射的维度。我们还提出了一个变体的UCRL RFE使用伯恩斯坦型奖金,并表明它需要采样最多$\tilde{\mathcal{O}}(H^4d(H + d)ε^{-2})$,以实现$ε$ -最优的政策。通过构造一类特殊的线性Mixture MDP,我们还证明了对于任何无报酬算法,至少需要对$\tilde Ω(H^2dε^{-2})$集进行采样才能得到$ε$ -最优策略.我们的上限匹配的$ε$和$d$的依赖性,如果$H \ge d$的下限。

英文摘要:

We study the model-based reward-free reinforcement learning with linear function approximation for episodic Markov decision processes (MDPs). In this setting, the agent works in two phases. In the exploration phase, the agent interacts with the environment and collects samples without the reward. In the planning phase, the agent is given a specific reward function and uses samples collected from the exploration phase to learn a good policy. We propose a new provably efficient algorithm, called UCRL-RFE under the Linear Mixture MDP assumption, where the transition probability kernel of the MDP can be parameterized by a linear function over certain feature mappings defined on the triplet of state, action, and next state. We show that to obtain an $ε$-optimal policy for arbitrary reward function, UCRL-RFE needs to sample at most $\tilde{\mathcal{O}}(H^5d^2ε^{-2})$ episodes during the exploration phase. Here, $H$ is the length of the episode, $d$ is the dimension of the feature mapping. We also propose a variant of UCRL-RFE using Bernstein-type bonus and show that it needs to sample at most $\tilde{\mathcal{O}}(H^4d(H + d)ε^{-2})$ to achieve an $ε$-optimal policy. By constructing a special class of linear Mixture MDPs, we also prove that for any reward-free algorithm, it needs to sample at least $\tilde Ω(H^2dε^{-2})$ episodes to obtain an $ε$-optimal policy. Our upper bound matches the lower bound in terms of the dependence on $ε$ and the dependence on $d$ if $H \ge d$.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑