arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2502.01591cs.LGcs.AI
收藏

改进基于Transformer的世界模型以实现数据高效强化学习

Improving Transformer World Models for Data-Efficient RL

Google DeepMind(谷歌DeepMind)

Antoine Dedieu, Joseph Ortiz, Xinghua Lou, Carter Wendelken, Wolfgang Lehrach, J Swaroop Guntupalli, Miguel Lazaro-Gredilla, Kevin Patrick Murphy

AI总结:

本文提出三项改进Transformer世界模型的技术——带预热的Dyna、最近邻分词器和分块教师强制,在Craftax-classic等基准上以1M环境步实现69.66%奖励,首次超越人类表现。

摘要:

我们对基于Transformer的标准模型化强化学习范式提出了三项改进:(a)“带预热的Dyna”,该方法在真实数据和想象数据上训练策略,但只有在世界模型经过充分训练后才开始使用想象数据;(b)用于图像块的“最近邻分词器”,它改进了以往的token化方案,这些方案在使用Transformer世界模型(TWM)时是必需的,通过确保码字在创建后保持静态,从而为TWM学习提供恒定目标;(c)“分块教师强制”,它允许TWM对下一时间步的未来token进行联合推理,而不是顺序生成它们。随后我们表明,我们的方法在各种环境中显著优于先前方法。我们主要聚焦于具有挑战性的Craftax-classic基准,在该基准上,我们的方法仅经过1M环境步就达到了69.66%的奖励,显著优于达到53.2%的DreamerV3,并首次超过人类65.0%的表现。我们还展示了在Craftax-full、MinAtar以及三个不同的双人游戏上的初步结果,以说明该方法的通用性。

英文摘要:

We present three improvements to the standard model-based RL paradigm based on transformers: (a) "Dyna with warmup", which trains the policy on real and imaginary data, but only starts using imaginary data after the world model has been sufficiently trained; (b) "nearest neighbor tokenizer" for image patches, which improves upon previous tokenization schemes, which are needed when using a transformer world model (TWM), by ensuring the code words are static after creation, thus providing a constant target for TWM learning; and (c) "block teacher forcing", which allows the TWM to reason jointly about the future tokens of the next timestep, instead of generating them sequentially. We then show that our method significantly improves upon prior methods in various environments. We mostly focus on the challenging Craftax-classic benchmark, where our method achieves a reward of 69.66% after only 1M environment steps, significantly outperforming DreamerV3, which achieves 53.2%, and exceeding human performance of 65.0% for the first time. We also show preliminary results on Craftax-full, MinAtar, and three different two-player games, to illustrate the generality of the approach.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑