通过预训练动力学表示实现任意演示的离线模仿学习
Offline Imitation Learning upon Arbitrary Demonstrations by Pre-Training Dynamics Representations
AI总结：
针对离线模仿学习有限数据瓶颈问题，提出预训练动力学表示的方法，理论证明最优决策变量位于表示空间以减少参数，可复用非专家数据，实验表明单轨迹即可模仿专家策略，还能从少量真实演示中学习行走。
摘要：
有限数据已成为扩展离线模仿学习（IL）的主要瓶颈。本文提出通过引入预训练阶段来提升有限专家数据下的IL性能，该阶段学习从转移动力学分解中导出的动力学表示。我们首先从理论上证明，离线IL的最优决策变量位于表示空间中，大幅减少了下游IL需学习的参数。此外，动力学表示可从具有相同动力学的任意数据中学习，允许复用大量非专家数据并缓解有限数据问题。我们提出一种受噪声对比估计启发的易处理损失函数，用于在预训练阶段学习动力学表示。在MuJoCo上的实验表明，所提算法仅需单条轨迹即可模仿专家策略；在真实四足机器人上的实验显示，可利用模拟器数据预训练的动力学表示，从少量真实世界演示中学习行走。
英文摘要：
Limited data has become a major bottleneck in scaling up offline imitation learning (IL). In this paper, we propose enhancing IL performance under limited expert data by introducing a pre-training stage that learns dynamics representations, derived from factorizations of the transition dynamics. We first theoretically justify that the optimal decision variable of offline IL lies in the representation space, significantly reducing the parameters to learn in the downstream IL. Moreover, the dynamics representations can be learned from arbitrary data collected with the same dynamics, allowing the reuse of massive non-expert data and mitigating the limited data issues. We present a tractable loss function inspired by noise contrastive estimation to learn the dynamics representations at the pre-training stage. Experiments on MuJoCo demonstrate that our proposed algorithm can mimic expert policies with as few as a single trajectory. Experiments on real quadrupeds show that we can leverage pre-trained dynamics representations from simulator data to learn to walk from a few real-world demonstrations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
