预测强化学习中的采样和时域效应
Effects of sampling and horizon in predictive reinforcement learning
摘要：
普通强化学习（RL）可能容易失去收敛，违反约束，意外的性能等，通常，RL代理经历广泛的学习阶段，以实现可接受的功能。这与典型地基于模型的经典控制算法形成对比。研究的一个方向是RL与这些算法的融合，特别是模型预测控制（MPC）。然而，这引入了与预测范围相关的新的超参数。此外，RL通常与马尔可夫决策过程有关。但大多数真实环境都不是时间离散的。强化学习的实际物理环境由一个数字主体和一个时间连续的动力系统组成。因此，事实上，还有另一个超参数-代理采样时间。在本文中，我们调查的预测范围和采样的两个混合RL-MPC-代理的移动机器人停车，这又是一个典型的控制问题的案例研究的影响。我们用MPC的一个简单变体对代理进行基准测试。抽样显示出一种“甜蜜点”的行为，而RL代理人在较短的视野表现出的优点。
英文摘要：
Plain reinforcement learning (RL) may be prone to loss of convergence, constraint violation, unexpected performance, etc. Commonly, RL agents undergo extensive learning stages to achieve acceptable functionality. This is in contrast to classical control algorithms which are typically model-based. An direction of research is the fusion of RL with such algorithms, especially model-predictive control (MPC). This, however, introduces new hyper-parameters related to the prediction horizon. Furthermore, RL is usually concerned with Markov decision processes. But the most of the real environments are not time-discrete. The factual physical setting of RL consists of a digital agent and a time-continuous dynamical system. There is thus, in fact, yet another hyper-parameter -- the agent sampling time. In this paper, we investigate the effects of prediction horizon and sampling of two hybrid RL-MPC-agents in a case study with a mobile robot parking, which is in turn a canonical control problem. We benchmark the agents with a simple variant of MPC. The sampling showed a kind of a "sweet spot" behavior, whereas the RL agents demonstrated merits at shorter horizons.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
