arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2103.04551cs.LG
收藏

来自虚空的行为:无监督的主动预训练

Behavior From the Void: Unsupervised Active Pre-Training

UC Berkeley(加州大学伯克利分校)

Hao Liu, Pieter Abbeel

摘要:

我们引入了一种新的无监督预训练方法用于强化学习,称为APT,代表主动预训练。APT通过在无奖励环境中主动搜索新状态来学习行为和表示。关键的新颖想法是通过最大化在抽象表示空间中计算的非参数熵来探索环境,这避免了具有挑战性的密度建模,因此允许我们的方法在具有高维观测的环境中更好地扩展(例如,图像观察)。我们通过在长时间无监督的预训练阶段后暴露特定于任务的奖励来经验性地评估APT。在Atari游戏中,APT在12个游戏中实现了人类水平的性能,与规范的全监督RL算法相比,获得了极具竞争力的性能。在DMControl套件上,APT在渐进性能和数据效率方面击败了所有基准,并显著提高了从头开始训练极其困难的任务的性能。

英文摘要:

We introduce a new unsupervised pre-training method for reinforcement learning called APT, which stands for Active Pre-Training. APT learns behaviors and representations by actively searching for novel states in reward-free environments. The key novel idea is to explore the environment by maximizing a non-parametric entropy computed in an abstract representation space, which avoids challenging density modeling and consequently allows our approach to scale much better in environments that have high-dimensional observations (e.g., image observations). We empirically evaluate APT by exposing task-specific reward after a long unsupervised pre-training phase. In Atari games, APT achieves human-level performance on 12 games and obtains highly competitive performance compared to canonical fully supervised RL algorithms. On DMControl suite, APT beats all baselines in terms of asymptotic performance and data efficiency and dramatically improves performance on tasks that are extremely difficult to train from scratch.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑