连接离线到在线强化学习的策略扩展
Policy Expansion for Bridging Offline-to-Online Reinforcement Learning
AI总结：
本文提出PEX策略扩展框架，通过保留离线策略并新增可学习策略进行自适应组合，缓解离线到在线强化学习微调初期的策略退化问题。
摘要：
利用离线数据进行预训练，并使用强化学习进行在线微调，是一种很有前景的学习控制策略的策略，可在样本效率和性能方面兼得两者之长。一种自然的方法是用离线训练得到的策略来初始化在线学习策略。在本工作中，我们为这一任务引入了一种策略扩展方案。在学到离线策略之后，我们将其作为策略集合中的一个候选策略。随后，我们用另一个负责进一步学习的策略来扩展该策略集合。这两个策略将以自适应方式组合，以与环境交互。通过这种方法，先前离线学到的策略在在线学习期间被完整保留，从而缓解在线学习初始阶段可能出现的破坏离线策略有用行为等潜在问题，同时允许离线策略以自适应方式自然参与探索。此外，新加入的策略有可能通过学习获得新的有用行为。我们在若干任务上进行了实验，结果证明了所提方法的有效性。代码可在 https://github.com/Haichao-Zhang/PEX 获取。
英文摘要：
Pre-training with offline data and online fine-tuning using reinforcement learning is a promising strategy for learning control policies by leveraging the best of both worlds in terms of sample efficiency and performance. One natural approach is to initialize the policy for online learning with the one trained offline. In this work, we introduce a policy expansion scheme for this task. After learning the offline policy, we use it as one candidate policy in a policy set. We then expand the policy set with another policy which will be responsible for further learning. The two policies will be composed in an adaptive manner for interacting with the environment. With this approach, the policy previously learned offline is fully retained during online learning, thus mitigating the potential issues such as destroying the useful behaviors of the offline policy in the initial stage of online learning while allowing the offline policy participate in the exploration naturally in an adaptive manner. Moreover, new useful behaviors can potentially be captured by the newly added policy through learning. Experiments are conducted on a number of tasks and the results demonstrate the effectiveness of the proposed approach. Code is available at https://github.com/Haichao-Zhang/PEX
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
