arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2302.06540cs.AIcs.LGcs.MA
收藏

自举对比学习的观察模仿

Imitation from Observation With Bootstrapped Contrastive Learning

University of Montreal(蒙特利尔大学);Mila(米拉研究所);McGill University(麦吉尔大学);Concordia University(康考迪亚大学)

Medric Sonwa, Johanna Hansen, Eugene Belilovsky

摘要:

观察模仿(IfO)是一种学习范式,它包括在马尔可夫决策过程(MDP)中通过观察专家演示而不访问其动作来训练自主智能体。这些演示可以是环境状态的序列或环境的原始视觉观察。IfO最近的工作集中在低维环境状态观测的情况下这个问题,然而,在实践中不太可能获得这些高度特定的观测。在本文中,我们采用了一个具有挑战性的,但更现实的问题制定,学习控制策略,操作的经验教训的潜在空间,只访问视觉演示的专家完成一项任务。我们提出了BootIfOL,IfO算法,旨在学习奖励函数,采取代理轨迹,并将其与专家进行比较,根据代理行为和隐式目标的相似性提供奖励。我们认为这个奖励函数是代理行为轨迹之间的距离度量,并通过对比学习来学习它。对比学习的目标是密切代表专家轨迹,并将它们从非专家轨迹。在对比学习中使用的非专家轨迹集通过使用当前奖励函数从RL学习的代理的推出进行自举而逐渐变得更加复杂。我们评估我们的方法在各种控制任务表明,我们可以训练有效的政策,使用有限数量的演示轨迹,大大提高了以前的方法,考虑原始观测。

英文摘要:

Imitation from observation (IfO) is a learning paradigm that consists of training autonomous agents in a Markov Decision Process (MDP) by observing expert demonstrations without access to its actions. These demonstrations could be sequences of environment states or raw visual observations of the environment. Recent work in IfO has focused on this problem in the case of observations of low-dimensional environment states, however, access to these highly-specific observations is unlikely in practice. In this paper, we adopt a challenging, but more realistic problem formulation, learning control policies that operate on a learned latent space with access only to visual demonstrations of an expert completing a task. We present BootIfOL, an IfO algorithm that aims to learn a reward function that takes an agent trajectory and compares it to an expert, providing rewards based on similarity to agent behavior and implicit goal. We consider this reward function to be a distance metric between trajectories of agent behavior and learn it via contrastive learning. The contrastive learning objective aims to closely represent expert trajectories and to distance them from non-expert trajectories. The set of non-expert trajectories used in contrastive learning is made progressively more complex by bootstrapping from roll-outs of the agent learned through RL using the current reward function. We evaluate our approach on a variety of control tasks showing that we can train effective policies using a limited number of demonstrative trajectories, greatly improving on prior approaches that consider raw observations.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑