arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2210.11604cs.LG
收藏

潜在马尔可夫决策过程的无视界且依赖方差的强化学习

Horizon-Free and Variance-Dependent Reinforcement Learning for Latent Markov Decision Processes

Paul G. Allen School of Computer Science & Engineering, University of Washington(华盛顿大学保罗·G·艾伦计算机科学与工程学院)

Runlong Zhou, Ruosong Wang, Simon S. Du

AI总结:

研究事后上下文LMDP的强化学习遗憾最小化问题,提出基于模型的乐观算法框架,首次实现近乎无视界且依赖问题的遗憾上界,并通过对称化技术构造下界证明其最优性。

摘要:

我们研究在具有事后上下文的潜在马尔可夫决策过程(LMDP)中进行强化学习(RL)的遗憾最小化问题。我们设计了一个新颖的基于模型的算法框架,该框架可以通过模型乐观求解器和价值乐观求解器进行实例化。我们证明了 $\tilde{O}(\sqrt{\mathsf{Var}^\star M ΓS A K})$ 的遗憾界,其中 $\tilde{O}$ 隐藏了对数因子,$M$ 是上下文的数量,$S$ 是状态的数量,$A$ 是动作的数量,$K$ 是回合的数量,$Γ\le S$ 是任何状态-动作对的最大转移度,而 $\mathsf{Var}^\star$ 是描述 LMDP 确定性的方差量。该遗憾界仅与规划视野呈对数缩放,从而产生了 LMDP 的第一个(近乎)无视界的遗憾界。这也是 LMDP 的第一个问题依赖的遗憾界。我们证明的关键是对 alpha 向量(价值函数的推广)的总方差分析,该分析通过截断方法处理。我们用一个新颖的 $Ω(\sqrt{\mathsf{Var}^\star M S A K})$ 遗憾下界(其中 $Γ= 2$)来补充我们的正面结果,这表明当 $Γ$ 对于有界方差 LMDP 类为常数时,我们的上界是极小化最优的。我们的下界依赖于困难实例的新构造以及受理论计算机科学对称化技术启发的论证,两者在技术上都与现有的 MDP 下界证明不同,因此可能具有独立的意义。

英文摘要:

We study regret minimization for reinforcement learning (RL) in Latent Markov Decision Processes (LMDPs) with context in hindsight. We design a novel model-based algorithmic framework which can be instantiated with both a model-optimistic and a value-optimistic solver. We prove an $\tilde{O}(\sqrt{\mathsf{Var}^\star M ΓS A K})$ regret bound where $\tilde{O}$ hides logarithm factors, $M$ is the number of contexts, $S$ is the number of states, $A$ is the number of actions, $K$ is the number of episodes, $Γ\le S$ is the maximum transition degree of any state-action pair, and $\mathsf{Var}^\star$ is a variance quantity describing the determinism of the LMDP. The regret bound only scales logarithmically with the planning horizon, thus yielding the first (nearly) horizon-free regret bound for LMDP. This is also the first problem-dependent regret bound for LMDP. Key in our proof is an analysis of the total variance of alpha vectors (a generalization of value functions), which is handled with a truncation method. We complement our positive result with a novel $Ω(\sqrt{\mathsf{Var}^\star M S A K})$ regret lower bound with $Γ= 2$, which shows our upper bound minimax optimal when $Γ$ is a constant for the class of variance-bounded LMDPs. Our lower bound relies on new constructions of hard instances and an argument inspired by the symmetrization technique from theoretical computer science, both of which are technically different from existing lower bound proof for MDPs, and thus can be of independent interest.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑