arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1906.04161cs.LGcs.AIcs.CVcs.ROstat.ML
收藏

通过分歧进行自我监督探索

Self-Supervised Exploration via Disagreement

CMU(卡内基梅隆大学);Facebook AI Research(Facebook人工智能研究院)

Deepak Pathak, Dhiraj Gandhi, Abhinav Gupta

摘要:

有效的探索是感觉运动学习中一个长期存在的问题。在无噪声、非随机的领域,如视频游戏和模拟,已经证明了重大进展。然而,这些公式中的大多数要么在具有随机动态的环境中卡住,要么效率太低,无法扩展到真正的机器人设置。在本文中,我们提出了一个公式的启发工作的探索主动学习文献。具体来说,我们训练一个动力学模型的集合,并激励代理进行探索,以使这些集合的分歧最大化。这允许代理通过以自我监督的方式探索来学习技能,而无需任何外部奖励。值得注意的是,我们进一步利用分歧目标以可区分的方式优化代理的策略,而不使用强化学习,这会导致样本有效的探索。我们证明了这种配方在各种基准环境,包括随机Atari,Mujoco和Unity的有效性。最后,我们实现了我们的微分探索一个真正的机器人,学会完全从头开始与对象进行交互。项目视频和代码在https://pathak22.github.io/exploration-by-disagreement/

英文摘要:

Efficient exploration is a long-standing problem in sensorimotor learning. Major advances have been demonstrated in noise-free, non-stochastic domains such as video games and simulation. However, most of these formulations either get stuck in environments with stochastic dynamics or are too inefficient to be scalable to real robotics setups. In this paper, we propose a formulation for exploration inspired by the work in active learning literature. Specifically, we train an ensemble of dynamics models and incentivize the agent to explore such that the disagreement of those ensembles is maximized. This allows the agent to learn skills by exploring in a self-supervised manner without any external reward. Notably, we further leverage the disagreement objective to optimize the agent's policy in a differentiable manner, without using reinforcement learning, which results in a sample-efficient exploration. We demonstrate the efficacy of this formulation across a variety of benchmark environments including stochastic-Atari, Mujoco and Unity. Finally, we implement our differentiable exploration on a real robot which learns to interact with objects completely from scratch. Project videos and code are at https://pathak22.github.io/exploration-by-disagreement/

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑