arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2010.12698cs.LG
收藏

基于Transformer的Q学习动作序列生成的稳定化

Stabilizing Transformer-Based Action Sequence Generation For Q-Learning

ITMO University(ITMO大学)

Gideon Stein, Andrey Filchenkov, Arip Asadulaev

摘要:

自最初的Transformer架构(Vaswani et al. 2017)发布以来,Transformers彻底改变了自然语言处理领域。这主要是因为它们比竞争对手基于RNN的架构更好地理解及时依赖关系的能力。令人惊讶的是,这种架构变化并没有影响强化学习(RL)领域,尽管RNN在RL中非常流行,并且时间依赖性在RL中非常常见。最近,Parisotto等人(2019)进行了RL中Transformers的第一项有希望的研究。为了支持这项工作的结果,本文试图提供一个基于transformer的强化学习方法的额外例子。具体来说,我们的目标是一个简单的基于transformer的深度Q学习方法,它在多种环境下都是稳定的。由于Transformers和RL的不稳定性,进行了广泛的方法搜索,以达到最终的方法,利用围绕Transformers以及Q学习的发展。所提出的方法可以匹配经典的Q-学习控制环境的性能,同时显示出一些选定的Atari基准的潜力。此外,它是批判性的评估,以提供更多的见解之间的关系Transformers和RL。

英文摘要:

Since the publication of the original Transformer architecture (Vaswani et al. 2017), Transformers revolutionized the field of Natural Language Processing. This, mainly due to their ability to understand timely dependencies better than competing RNN-based architectures. Surprisingly, this architecture change does not affect the field of Reinforcement Learning (RL), even though RNNs are quite popular in RL, and time dependencies are very common in RL. Recently, Parisotto et al. 2019) conducted the first promising research of Transformers in RL. To support the findings of this work, this paper seeks to provide an additional example of a Transformer-based RL method. Specifically, the goal is a simple Transformer-based Deep Q-Learning method that is stable over several environments. Due to the unstable nature of Transformers and RL, an extensive method search was conducted to arrive at a final method that leverages developments around Transformers as well as Q-learning. The proposed method can match the performance of classic Q-learning on control environments while showing potential on some selected Atari benchmarks. Furthermore, it was critically evaluated to give additional insights into the relation between Transformers and RL.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑