arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2303.01668cs.LGcs.AI
收藏

RePreM:基于掩蔽模型的强化学习表示预训练

RePreM: Representation Pre-training with Masked Model for Reinforcement Learning

Tsinghua University(清华大学);Microsoft Research Asia(微软亚洲研究院);Hulu;Macquarie University(麦考瑞大学);University of Exeter(埃克塞特大学)

Yuanying Cai, Chuheng Zhang, Wei Shen, Xuyun Zhang, Wenjie Ruan, Longbo Huang

摘要:

受最近RL中序列建模的成功和使用掩蔽语言模型进行预训练的启发,我们提出了一种用于RL中预训练的掩蔽模型RePreM(Masked Model的表示预训练),它训练编码器结合Transformer块来预测轨迹中的掩蔽状态或动作。与RL中现有的表示预训练方法相比,RePreM简单但有效。它避免了序列建模的算法复杂性(如数据增强或估计多个模型),并生成一个能够很好地捕捉长期动态的表示。从经验上讲,我们证明了RePreM在各种任务中的有效性,包括动态预测,迁移学习和基于值和演员-评论家方法的样本高效RL。此外,我们还证明了RePreM与数据集大小、数据集质量和编码器规模的关系很好,这表明了它对大型RL模型的潜力。

英文摘要:

Inspired by the recent success of sequence modeling in RL and the use of masked language model for pre-training, we propose a masked model for pre-training in RL, RePreM (Representation Pre-training with Masked Model), which trains the encoder combined with transformer blocks to predict the masked states or actions in a trajectory. RePreM is simple but effective compared to existing representation pre-training methods in RL. It avoids algorithmic sophistication (such as data augmentation or estimating multiple models) with sequence modeling and generates a representation that captures long-term dynamics well. Empirically, we demonstrate the effectiveness of RePreM in various tasks, including dynamic prediction, transfer learning, and sample-efficient RL with both value-based and actor-critic methods. Moreover, we show that RePreM scales well with dataset size, dataset quality, and the scale of the encoder, which indicates its potential towards big RL models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑