arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2305.16985cs.LG
收藏

逆向动力学预训练学习适用于多任务模仿的良好表示

Inverse Dynamics Pretraining Learns Good Representations for Multitask Imitation

New York University(纽约大学);Google(谷歌)

David Brandfonbrener, Ofir Nachum, Joan Bruna

AI总结:

本研究探讨了模仿学习中的预训练范式,提出逆向动力学建模作为多任务预训练目标,学习可迁移的低维观测表示,并通过模拟实验和理论分析验证了其有效性。

摘要:

近年来,自然语言处理和图像识别等领域推广了使用大规模数据集预训练表示的范式,这些表示可以有效迁移到下游任务中。在本研究中,我们评估了这种范式在模仿学习中应如何实施,其中预训练和微调数据都是由专家与未知环境交互所收集的轨迹。即,我们考虑一种设置,其中预训练语料库由多任务演示组成,每个演示的任务由未观测到的潜在上下文变量设定。目标是使用预训练语料库学习高维(例如视觉)观测空间的低维表示,该表示可以迁移到新的上下文,以便在有限的演示数据集上进行微调。在各种可能的预训练目标中,我们认为逆向动力学建模——即在演示中给定动作前后的观测来预测该动作——非常适合这种设置。我们通过对各种模拟视觉运动操作问题的评估,为这一主张提供了实证证据。虽然之前的工作试图对逆向动力学建模的优势进行各种理论解释,但我们发现这些论点不足以解释在我们的设置中经常观察到的经验优势,因此我们使用一个简单但通用的环境模型推导出了一种新的分析。

英文摘要:

In recent years, domains such as natural language processing and image recognition have popularized the paradigm of using large datasets to pretrain representations that can be effectively transferred to downstream tasks. In this work we evaluate how such a paradigm should be done in imitation learning, where both pretraining and finetuning data are trajectories collected by experts interacting with an unknown environment. Namely, we consider a setting where the pretraining corpus consists of multitask demonstrations and the task for each demonstration is set by an unobserved latent context variable. The goal is to use the pretraining corpus to learn a low dimensional representation of the high dimensional (e.g., visual) observation space which can be transferred to a novel context for finetuning on a limited dataset of demonstrations. Among a variety of possible pretraining objectives, we argue that inverse dynamics modeling -- i.e., predicting an action given the observations appearing before and after it in the demonstration -- is well-suited to this setting. We provide empirical evidence of this claim through evaluations on a variety of simulated visuomotor manipulation problems. While previous work has attempted various theoretical explanations regarding the benefit of inverse dynamics modeling, we find that these arguments are insufficient to explain the empirical advantages often observed in our settings, and so we derive a novel analysis using a simple but general environment model.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑