arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2205.06595stat.MLcs.AIcs.LG
收藏

倒置强化学习可以在具有情节重置的随机环境中发散

Upside-Down Reinforcement Learning Can Diverge in Stochastic Environments With Episodic Resets

The Swiss AI Lab IDSIA(瑞士人工智能实验室IDSIA);Università della Svizzera italiana (USI)(瑞士意大利语区大学);Scuola universitaria professionale della Svizzera italiana (SUPSI)(瑞士意大利语区应用科学大学);King Abdullah University of Science and Technology (KAUST)(阿卜杜拉国王科技大学);NNAISENSE

Miroslav Štrupl, Francesco Faccio, Dylan R. Ashley, Jürgen Schmidhuber, Rupesh Kumar Srivastava

摘要:

倒置强化学习(UDRL)是一种解决强化学习问题的方法,它不需要值函数,只使用监督学习,其中数据集中给定输入的目标不会随着时间的推移而改变。Ghosh等人证明了目标条件监督学习(GCSL)-可以被视为UDRL的简化版本-优化了目标达成性能的下限。这提高了这样的算法可以在任意环境中保证收敛到最优策略的期望,类似于某些众所周知的传统RL算法。在这里,我们表明,对于一个特定的情节UDRL算法(eUDRL,包括GCSL),这是不是这种情况下,并给出这种限制的原因。为此,我们首先引入了一个有用的eUDRL重写作为递归策略更新。这种提法有助于反驳其收敛到最优策略的一类随机环境。最后,我们提供了一个非常简单的eUDRL发散环境的具体示例。由于本文的主要目的是提出一个否定的结果,最好的反例是最简单的,我们限制所有的讨论有限(离散)的环境,忽略函数逼近和有限的样本量的问题。

英文摘要:

Upside-Down Reinforcement Learning (UDRL) is an approach for solving RL problems that does not require value functions and uses only supervised learning, where the targets for given inputs in a dataset do not change over time. Ghosh et al. proved that Goal-Conditional Supervised Learning (GCSL) -- which can be viewed as a simplified version of UDRL -- optimizes a lower bound on goal-reaching performance. This raises expectations that such algorithms may enjoy guaranteed convergence to the optimal policy in arbitrary environments, similar to certain well-known traditional RL algorithms. Here we show that for a specific episodic UDRL algorithm (eUDRL, including GCSL), this is not the case, and give the causes of this limitation. To do so, we first introduce a helpful rewrite of eUDRL as a recursive policy update. This formulation helps to disprove its convergence to the optimal policy for a wide class of stochastic environments. Finally, we provide a concrete example of a very simple environment where eUDRL diverges. Since the primary aim of this paper is to present a negative result, and the best counterexamples are the simplest ones, we restrict all discussions to finite (discrete) environments, ignoring issues of function approximation and limited sample size.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑