后验行为克隆：为高效强化学习微调预训练BC策略
Posterior Behavioral Cloning: Pretraining BC Policies for Efficient RL Finetuning
AI总结：
本文提出后验行为克隆策略，通过建模示范者行为的后验分布来提升强化学习微调效果。
摘要：
在从机器人到语言学的各个领域中，标准做法是首先在大规模示范数据集上预训练一个策略，然后通过强化学习（RL）进行微调，以提高在部署领域中的性能。这个微调步骤在实现人类或超人类性能中已被证明是关键的，然而尽管已经很多关注被放在开发更有效的微调算法上，却很少关注确保预训练策略是RL微调的有效初始化。在本文中，我们试图了解预训练策略如何影响微调性能，以及如何预训练策略以确保它们是微调的有效初始化。我们首先理论上表明标准行为克隆（BC）——它训练策略直接匹配示范者的行为——无法确保覆盖示范者的行为，这是有效RL微调的最小必要条件。然后我们显示，如果而不是精确拟合观察到的示范，我们训练策略以建模给定示范数据集的示范者行为的后验分布，我们确实获得了一个确保覆盖示范者行为的策略，从而实现更有效的微调。此外，这种策略——我们称之为后验行为克隆（PostBC）策略——在确保预训练性能不低于BC策略的情况下实现了这一点。然后我们显示PostBC可以使用现代生成模型在机器人控制领域中实际实现——仅依赖标准监督学习，并且在现实的机器人控制基准和真实世界的机器人操作任务中，与标准行为克隆相比，导致显著改进的RL微调性能。
英文摘要：
Standard practice across domains from robotics to language is to first pretrain a policy on a large-scale demonstration dataset, and then finetune this policy, typically with reinforcement learning (RL), in order to improve performance on deployment domains. This finetuning step has proved critical in achieving human or super-human performance, yet while much attention has been given to developing more effective finetuning algorithms, little attention has been given to ensuring the pretrained policy is an effective initialization for RL finetuning. In this work we seek to understand how the pretrained policy affects finetuning performance, and how to pretrain policies in order to ensure they are effective initializations for finetuning. We first show theoretically that standard behavioral cloning (BC) -- which trains a policy to directly match the actions played by the demonstrator -- can fail to ensure coverage over the demonstrator's actions, a minimal condition necessary for effective RL finetuning. We then show that if, instead of exactly fitting the observed demonstrations, we train a policy to model the posterior distribution of the demonstrator's behavior given the demonstration dataset, we do obtain a policy that ensures coverage over the demonstrator's actions, enabling more effective finetuning. Furthermore, this policy -- which we refer to as the posterior behavioral cloning (PostBC) policy -- achieves this while ensuring pretrained performance is no worse than that of the BC policy. We then show that PostBC is practically implementable with modern generative models in robotic control domains -- relying only on standard supervised learning -- and leads to significantly improved RL finetuning performance on both realistic robotic control benchmarks and real-world robotic manipulation tasks, as compared to standard behavioral cloning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
