Imitation from observation (IfO) is a learning paradigm that consists of training autonomous agents in a Markov Decision Process (MDP) by observing expert demonstrations without access to its actions. These demonstrations could be sequences of environment states or raw visual observations of the environment. Recent work in IfO has focused on this problem in the case of observations of low-dimensional environment states, however, access to these highly-specific observations is unlikely in practice. In this paper, we adopt a challenging, but more realistic problem formulation, learning control policies that operate on a learned latent space with access only to visual demonstrations of an expert completing a task. We present BootIfOL, an IfO algorithm that aims to learn a reward function that takes an agent trajectory and compares it to an expert, providing rewards based on similarity to agent behavior and implicit goal. We consider this reward function to be a distance metric between trajectories of agent behavior and learn it via contrastive learning. The contrastive learning objective aims to closely represent expert trajectories and to distance them from non-expert trajectories. The set of non-expert trajectories used in contrastive learning is made progressively more complex by bootstrapping from roll-outs of the agent learned through RL using the current reward function. We evaluate our approach on a variety of control tasks showing that we can train effective policies using a limited number of demonstrative trajectories, greatly improving on prior approaches that consider raw observations.
University of Tsinghua(清华大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Shanghai Qi Zhi Institute(上海期智研究院);Hong Kong University of Science and Technology(香港科技大学)
本文提出策略对比模仿学习(PCIL),通过锚定不同策略学习对比表示空间并生成平滑余弦相似度奖励,以解决 AIL 判别器表示质量低的问题,在 DeepMind Control suite 上取得最先进性能。
Seek Before You Move: Evidence Seeking for Progress Grounding in Vision-Language Navigation
Tsinghua University(清华大学);Pengcheng Laboratory(鹏城实验室);South China University of Technology(华南理工大学);International Digital Economy Academy(国际数字经济学院);Ping An Technology (Shenzhen) Co., Ltd.(平安科技(深圳)有限公司);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳))