arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2211.13337cs.LG
收藏

多环境预训练支持将有限数据集传输到行动

Multi-Environment Pretraining Enables Transfer to Action Limited Datasets

University of California, Berkeley(加州大学伯克利分校);Google Brain(谷歌大脑);McGill University(麦吉尔大学);Mila(米拉人工智能研究院);DeepMind(深度思维)

David Venuto, Sherry Yang, Pieter Abbeel, Doina Precup, Igor Mordatch, Ofir Nachum

摘要:

使用大规模数据集来训练大规模模型已经成为自然语言和视觉应用中广泛泛化的主要方法。然而,在强化学习中,一个关键的挑战是顺序决策的可用数据通常没有用动作来注释-例如,玩游戏的视频比与其记录的游戏控制配对的帧序列更可用。我们建议规避这一挑战相结合的大型,但稀疏注释的数据集从\emph{target}环境的利益与完全注释的数据集从各种其他\emph{source}环境。我们的方法,行动有限预训练(ALPT),利用逆动力学建模(IDM)的泛化能力,在目标环境中标记缺失的动作数据。我们表明,在IDM预训练过程中,即使利用一个额外的环境数据集的标记数据,也会在生成未注释序列的动作标签方面产生实质性的改进。我们评估我们的方法在基准游戏环境,并表明,我们可以显着提高游戏性能和泛化能力相比,其他方法,使用注释数据集相当于只有$12$分钟的游戏。突出IDM的力量,我们表明,即使目标和源环境没有共同的行动,这些好处仍然存在。

英文摘要:

Using massive datasets to train large-scale models has emerged as a dominant approach for broad generalization in natural language and vision applications. In reinforcement learning, however, a key challenge is that available data of sequential decision making is often not annotated with actions - for example, videos of game-play are much more available than sequences of frames paired with their logged game controls. We propose to circumvent this challenge by combining large but sparsely-annotated datasets from a \emph{target} environment of interest with fully-annotated datasets from various other \emph{source} environments. Our method, Action Limited PreTraining (ALPT), leverages the generalization capabilities of inverse dynamics modelling (IDM) to label missing action data in the target environment. We show that utilizing even one additional environment dataset of labelled data during IDM pretraining gives rise to substantial improvements in generating action labels for unannotated sequences. We evaluate our method on benchmark game-playing environments and show that we can significantly improve game performance and generalization capability compared to other approaches, using annotated datasets equivalent to only $12$ minutes of gameplay. Highlighting the power of IDM, we show that these benefits remain even when target and source environments share no common actions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑