比较微调与元学习在少样本策略模仿中的有效性
Comparing the Efficacy of Fine-Tuning and Meta-Learning for Few-Shot Policy Imitation
AI总结：
本文比较微调与元学习在少样本策略模仿中的效果，提出基于强化学习预训练和行为克隆微调的两阶段基线，并发布iMuJoCo数据集，证明其简单且有效。
摘要：
在本文中，我们探讨了控制问题的少样本模仿学习，该问题涉及通过访问有限数量的离线轨迹来学习模仿目标策略。尽管这一设置与机器人和控制应用高度相关，但相关研究相对不足。当前用于解决少样本模仿的最先进方法依赖于元学习，而元学习训练成本高昂，因为它需要访问任务分布（来自许多目标策略和基础环境变体的轨迹）。鉴于这一局限性，我们研究了一种替代方法——微调，这是一类先在单一数据集上预训练，再在未见过的领域特定数据上进行微调的方法。近期研究表明，在少样本图像分类任务中，微调方法优于元学习方法，尤其是在数据分布外的情况下。在此，我们评估了这一点在控制问题上的适用程度，并提出了一种简单而有效的基线方法，该方法分为两个阶段：（i）通过强化学习（如Soft Actor-Critic）在单一基础环境上在线训练基础策略；（ii）通过行为克隆在目标策略的少量离线轨迹上微调基础策略。尽管该基线方法简单，但在多种条件下与元学习方法具有竞争力，并且能够模仿在原始环境未见变体上训练的目标策略。重要的是，所提出的方法实用且易于实现，因为它不需要任何复杂的元训练协议。作为进一步贡献，我们发布了一个名为iMuJoCo（iMitation MuJoCo）的开源数据集，包含154个流行的OpenAI-Gym MuJoCo环境变体，以及相关的预训练目标策略和轨迹，可供社区用于研究少样本模仿学习和离线强化学习。
英文摘要：
In this paper we explore few-shot imitation learning for control problems, which involves learning to imitate a target policy by accessing a limited set of offline rollouts. This setting has been relatively under-explored despite its relevance to robotics and control applications. State-of-the-art methods developed to tackle few-shot imitation rely on meta-learning, which is expensive to train as it requires access to a distribution over tasks (rollouts from many target policies and variations of the base environment). Given this limitation we investigate an alternative approach, fine-tuning, a family of methods that pretrain on a single dataset and then fine-tune on unseen domain-specific data. Recent work has shown that fine-tuners outperform meta-learners in few-shot image classification tasks, especially when the data is out-of-domain. Here we evaluate to what extent this is true for control problems, proposing a simple yet effective baseline which relies on two stages: (i) training a base policy online via reinforcement learning (e.g. Soft Actor-Critic) on a single base environment, (ii) fine-tuning the base policy via behavioral cloning on a few offline rollouts of the target policy. Despite its simplicity this baseline is competitive with meta-learning methods on a variety of conditions and is able to imitate target policies trained on unseen variations of the original environment. Importantly, the proposed approach is practical and easy to implement, as it does not need any complex meta-training protocol. As a further contribution, we release an open source dataset called iMuJoCo (iMitation MuJoCo) consisting of 154 variants of popular OpenAI-Gym MuJoCo environments with associated pretrained target policies and rollouts, which can be used by the community to study few-shot imitation learning and offline reinforcement learning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
