基于模型的强化学习作为实现无视界与二阶界的极简方法
Model-based RL as a Minimalist Approach to Horizon-Free and Second-Order Bounds
AI总结：
本研究证明，配备乐观/悲观规划的MLE基础版基于模型RL框架，可在时齐、奖励归一化条件下实现近乎无视界的二阶遗憾与样本复杂度界，算法简洁且易适配非线性函数近似。
摘要：
通过极大似然估计（MLE）学习转移模型，再在习得模型内进行规划，或许是最标准、最简单的基于模型的强化学习（RL）框架。在本研究中，我们证明这种简单的基于模型的RL方案在配备乐观与悲观规划流程后，能在在线和离线RL场景中取得优异的遗憾界和样本复杂度界。\n具体而言，我们证明在轨迹级奖励归一化至0到1之间、转移具有时齐性的条件下，该方案能实现近乎无视界的二阶界。近乎无视界指我们的界与马尔可夫决策过程的视界不存在多项式依赖关系。二阶界是一种依赖于实例的界，其规模随策略回报的方差变化：当系统近乎确定，且（或）最优策略的取值较小时，该方差可能很小。\n我们强调，所提出的算法简单且相当标准，实际上已在RL文献中被广泛研究：它们通过MLE学习模型，围绕MLE解构建版本空间，并根据工作在在线还是离线模式执行乐观或悲观规划。这些算法不依赖额外的专门算法设计（如学习方差、执行方差加权学习），因此可轻松利用非线性函数近似。算法的简洁性也意味着，我们的无视界二阶遗憾分析实则具备标准性，主要遵循面对不确定性时的乐观/悲观通用框架。
英文摘要：
Learning a transition model via Maximum Likelihood Estimation (MLE) followed by planning inside the learned model is perhaps the most standard and simplest Model-based Reinforcement Learning (RL) framework. In this work, we show that such a simple Model-based RL scheme, when equipped with optimistic and pessimistic planning procedures, achieves strong regret and sample complexity bounds in online and offline RL settings. Particularly, we demonstrate that under the conditions where the trajectory-wise reward is normalized between zero and one and the transition is time-homogenous, it achieves nearly horizon-free and second-order bounds. Nearly horizon-free means that our bounds have no polynomial dependence on the horizon of the Markov Decision Process. A second-order bound is a type of instance-dependent bound that scales with respect to the variances of the returns of the policies which can be small when the system is nearly deterministic and (or) the optimal policy has small values. We highlight that our algorithms are simple, fairly standard, and indeed have been extensively studied in the RL literature: they learn a model via MLE, build a version space around the MLE solution, and perform optimistic or pessimistic planning depending on whether operating in the online or offline mode. These algorithms do not rely on additional specialized algorithmic designs such as learning variances and performing variance-weighted learning and thus can easily leverage non-linear function approximations. The simplicity of the algorithms also implies that our horizon-free and second-order regret analysis is actually standard and mainly follows the general framework of optimism/pessimism in the face of uncertainty.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
