基于网格的低分形维数强化学习策略分析
Mesh Based Analysis of Low Fractal Dimension Reinforcement Learning Policies
摘要：
在以前的工作中，使用一个过程，我们称之为网格，可达状态空间的各种连续和混合动力系统近似为一个离散的状态集，然后可以合成到一个马尔可夫链。这种方法的应用之一是分析通过强化学习获得的运动策略，这是对所得系统的稳定性进行经验保证的一步。在另一条研究路线中，我们为基于策略的强化学习算法引入了一个修改后的奖励函数，该算法利用了滚动轨迹的“分形维数”。这种奖励被证明是鼓励政策，诱导个人的轨迹，可以更复杂地表示为离散网格。在这项工作中，我们结合这两个线程的研究，建立网格的可达状态空间的系统受到干扰和控制的政策与修改后的奖励。我们的分析表明，修改后的政策产生更小的可达网格。这表明，用分形维数奖励训练的代理将其具有更紧凑状态空间的理想质量转移到具有外部干扰的设置中。结果还表明，以前的工作使用基于网格的工具来分析RL策略可以扩展到更高维的系统或更高分辨率的网格，否则是可能的。
英文摘要：
In previous work, using a process we call meshing, the reachable state spaces for various continuous and hybrid systems were approximated as a discrete set of states which can then be synthesized into a Markov chain. One of the applications for this approach has been to analyze locomotion policies obtained by reinforcement learning, in a step towards making empirical guarantees about the stability properties of the resulting system. In a separate line of research, we introduced a modified reward function for on-policy reinforcement learning algorithms that utilizes a "fractal dimension" of rollout trajectories. This reward was shown to encourage policies that induce individual trajectories which can be more compactly represented as a discrete mesh. In this work we combine these two threads of research by building meshes of the reachable state space of a system subject to disturbances and controlled by policies obtained with the modified reward. Our analysis shows that the modified policies do produce much smaller reachable meshes. This shows that agents trained with the fractal dimension reward transfer their desirable quality of having a more compact state space to a setting with external disturbances. The results also suggest that the previous work using mesh based tools to analyze RL policies may be extended to higher dimensional systems or to higher resolution meshes than would have otherwise been possible.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
