世界模型骨干网络的对决:RNN、Transformer与S4
Facing Off World Model Backbones: RNNs, Transformers, and S4
AI总结:
针对现有MBRL世界模型骨干RNN记忆容量有限的问题,提出兼容可并行SSM的S4WM世界模型,实验证明其在长期记忆任务上优于Transformer类模型,且训练和推理效率更高。
摘要:
世界模型是基于模型的强化学习(MBRL)中的核心组件。要在部分可观测环境中对未来进行时间跨度较长且具备一致性的模拟,世界模型需要拥有长期记忆能力。但目前最先进的MBRL智能体,例如Dreamer,大多采用循环神经网络(RNNs)作为世界模型的骨干网络,这类网络的记忆容量十分有限。\n本文旨在探索可提升长期记忆能力的替代型世界模型骨干网络。我们重点研究了Transformer与结构化状态空间序列(Structured State Space Sequence,S4)模型的有效性,研究动机在于二者在捕捉低维序列长程依赖方面的出色表现,以及二者具备的互补优势。我们提出了S4WM,这是首个可与包括S4及其变体在内的可并行化状态空间模型(SSMs)兼容的世界模型。通过引入潜变量建模,S4WM能够通过潜想象高效生成高维图像序列。\n此外,我们在四组定制环境中对基于RNN、Transformer和S4的世界模型进行了全面对比,这些环境专门用于评估世界模型的关键记忆能力,包括长期想象、上下文依赖召回、奖励预测以及基于记忆的推理。研究结果表明,S4WM在长期记忆方面的表现优于基于Transformer的世界模型,同时在训练和想象过程中具备更高的效率。这些结果为开发性能更强的MBRL智能体铺平了道路。
英文摘要:
World models are a fundamental component in model-based reinforcement learning (MBRL). To perform temporally extended and consistent simulations of the future in partially observable environments, world models need to possess long-term memory. However, state-of-the-art MBRL agents, such as Dreamer, predominantly employ recurrent neural networks (RNNs) as their world model backbone, which have limited memory capacity. In this paper, we seek to explore alternative world model backbones for improving long-term memory. In particular, we investigate the effectiveness of Transformers and Structured State Space Sequence (S4) models, motivated by their remarkable ability to capture long-range dependencies in low-dimensional sequences and their complementary strengths. We propose S4WM, the first world model compatible with parallelizable SSMs including S4 and its variants. By incorporating latent variable modeling, S4WM can efficiently generate high-dimensional image sequences through latent imagination. Furthermore, we extensively compare RNN-, Transformer-, and S4-based world models across four sets of environments, which we have tailored to assess crucial memory capabilities of world models, including long-term imagination, context-dependent recall, reward prediction, and memory-based reasoning. Our findings demonstrate that S4WM outperforms Transformer-based world models in terms of long-term memory, while exhibiting greater efficiency during training and imagination. These results pave the way for the development of stronger MBRL agents.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。