自最初的Transformer架构(Vaswani et al. 2017)发布以来,Transformers彻底改变了自然语言处理领域。这主要是因为它们比竞争对手基于RNN的架构更好地理解及时依赖关系的能力。令人惊讶的是,这种架构变化并没有影响强化学习(RL)领域,尽管RNN在RL中非常流行,并且时间依赖性在RL中非常常见。最近,Parisotto等人(2019)进行了RL中Transformers的第一项有希望的研究。为了支持这项工作的结果,本文试图提供一个基于transformer的强化学习方法的额外例子。具体来说,我们的目标是一个简单的基于transformer的深度Q学习方法,它在多种环境下都是稳定的。由于Transformers和RL的不稳定性,进行了广泛的方法搜索,以达到最终的方法,利用围绕Transformers以及Q学习的发展。所提出的方法可以匹配经典的Q-学习控制环境的性能,同时显示出一些选定的Atari基准的潜力。此外,它是批判性的评估,以提供更多的见解之间的关系Transformers和RL。
英文摘要:
Since the publication of the original Transformer architecture (Vaswani et al. 2017), Transformers revolutionized the field of Natural Language Processing. This, mainly due to their ability to understand timely dependencies better than competing RNN-based architectures. Surprisingly, this architecture change does not affect the field of Reinforcement Learning (RL), even though RNNs are quite popular in RL, and time dependencies are very common in RL. Recently, Parisotto et al. 2019) conducted the first promising research of Transformers in RL. To support the findings of this work, this paper seeks to provide an additional example of a Transformer-based RL method. Specifically, the goal is a simple Transformer-based Deep Q-Learning method that is stable over several environments. Due to the unstable nature of Transformers and RL, an extensive method search was conducted to arrive at a final method that leverages developments around Transformers as well as Q-learning. The proposed method can match the performance of classic Q-learning on control environments while showing potential on some selected Atari benchmarks. Furthermore, it was critically evaluated to give additional insights into the relation between Transformers and RL.
Stabilizing Voltage in Power Distribution Networks via Multi-Agent Reinforcement Learning with Transformer
University of Science and Technology of China(中国科学技术大学);Institute of Artificial Intelligence, Hefei Comprehensive National Science Center(合肥综合性国家科学中心人工智能研究院)
Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentation
University of California, San Diego(加利福尼亚大学圣迭戈分校)
虽然通过强化学习(RL)训练的智能体可以直接从视觉观察中解决越来越具有挑战性的任务,但将学习到的技能推广到新的环境仍然非常具有挑战性。广泛使用数据增强是一种很有前途的提高RL泛化能力的技术,但它通常会降低样本效率,甚至会导致发散。在本文中,我们调查的不稳定性的原因时,使用数据增强在常见的非政策RL算法。我们确定了两个问题,都植根于高方差Q目标。基于我们的研究结果,我们提出了一个简单而有效的技术,稳定这类算法下增强。我们使用ConvNets和视觉Transformer(ViT)对基于DeepMind Control Suite的一系列基准以及机器人操作任务进行了广泛的基于图像的RL的实证评估。我们的方法大大提高了增强条件下ConvNets的稳定性和样本效率,并在具有不可见视觉效果的环境中实现了与基于图像的RL的最先进方法竞争的泛化结果。我们进一步表明,我们的方法可扩展到基于ViT的架构的RL,并且数据增强在这种情况下可能特别重要。
StARformer: Transformer with State-Action-Reward Representations for Visual Reinforcement Learning
Stony Brook University(石溪大学)
强化学习(RL)可以被认为是一个序列建模任务:给定一系列过去的状态-动作-奖励经验,智能体预测一系列接下来的动作。在这项工作中,我们提出了视觉RL的状态-动作-奖励Transformer(StARformer),它显式地模拟短期状态-动作-奖励表示(StAR表示),本质上是引入了类似马尔可夫的归纳偏差,以改善长期建模。我们的方法首先提取星表示的自我参加图像状态补丁,动作和奖励令牌在一个很短的时间窗口。然后将这些与纯图像状态表示相结合-提取为卷积特征,以在整个序列上执行自我注意。我们的实验表明,在离线RL和模仿学习设置中,StARformer在基于图像的Atari和DeepMind Control Suite基准测试中的表现优于最先进的基于Transformer的方法。StARformer也更符合较长的输入序列。我们的代码可在https://github.com/elicassion/StARformer上获得。
Action Transformer: A Self-Attention Model for Short-Time Pose-Based Human Action Recognition
Politecnico di Torino(都灵理工大学);PIC4SeR, Politecnico di Torino Interdepartmental Centre for Service Robotics(都灵理工大学跨部门服务机器人中心 PIC4SeR);Newcastle University(纽卡斯尔大学)