arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2006.07178cs.LGstat.ML
收藏

基于模型辨识和经验重标记的Meta-强化学习对分布迁移的鲁棒性

Meta-Reinforcement Learning Robust to Distributional Shift via Model Identification and Experience Relabeling

Stanford University(斯坦福大学);University of California, Berkeley(加州大学伯克利分校)

Russell Mendonca, Xinyang Geng, Chelsea Finn, Sergey Levine

摘要:

强化学习算法可以自主地获取复杂任务的策略。然而,学习一组不同的技能所需的样本数量可能非常大。虽然Meta强化学习方法使智能体能够利用先前的经验快速适应新任务,但它们的性能关键取决于新任务与先前经历的任务的接近程度。目前的方法要么不能很好地外推,要么可以以需要非常大量的数据进行策略元训练为代价。在这项工作中,我们提出了模型识别和经验重新标记(MIER),一个Meta强化学习算法,既有效,外推以及当面临的分布任务在测试时。我们的方法是基于一个简单的见解:我们认识到,动态模型可以有效地适应和一致的政策外的数据,更容易比政策和价值函数。然后,这些动态模型可以用于通过为新任务生成合成经验来继续训练分发外任务的策略和价值函数,而根本不使用Meta强化学习。

英文摘要:

Reinforcement learning algorithms can acquire policies for complex tasks autonomously. However, the number of samples required to learn a diverse set of skills can be prohibitively large. While meta-reinforcement learning methods have enabled agents to leverage prior experience to adapt quickly to new tasks, their performance depends crucially on how close the new task is to the previously experienced tasks. Current approaches are either not able to extrapolate well, or can do so at the expense of requiring extremely large amounts of data for on-policy meta-training. In this work, we present model identification and experience relabeling (MIER), a meta-reinforcement learning algorithm that is both efficient and extrapolates well when faced with out-of-distribution tasks at test time. Our method is based on a simple insight: we recognize that dynamics models can be adapted efficiently and consistently with off-policy data, more easily than policies and value functions. These dynamics models can then be used to continue training policies and value functions for out-of-distribution tasks without using meta-reinforcement learning at all, by generating synthetic experience for the new task.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑