arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1802.10592cs.LGcs.AIcs.RO
收藏

模型包围信任域策略优化

Model-Ensemble Trust-Region Policy Optimization

Berkeley AI Research(伯克利人工智能研究中心);University of California, Berkeley(加州大学伯克利分校)

Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, Pieter Abbeel

摘要:

无模型强化学习(RL)方法在越来越多的任务中取得了成功,这得益于深度学习的最新进展。然而,它们往往遭受高样本复杂性,这阻碍了它们在现实世界中的使用。或者,基于模型的强化学习有望降低样本复杂性,但往往需要仔细调整,迄今为止主要在简单模型足以进行学习的限制性领域取得了成功。在本文中,我们分析了当使用深度神经网络学习模型和策略时,香草基于模型的强化学习方法的行为,并表明学习的策略倾向于利用数据不足的区域来学习模型,从而导致训练不稳定。为了克服这个问题,我们建议使用一个集成的模型来保持模型的不确定性和正规化的学习过程。我们进一步表明,使用似然比导数产生更稳定的学习比反向传播通过时间。总而言之,我们的方法模型包围信任区域策略优化(ME-TRPO)与无模型深度RL方法相比,在具有挑战性的连续控制基准任务上显着降低了样本复杂度。

英文摘要:

Model-free reinforcement learning (RL) methods are succeeding in a growing number of tasks, aided by recent advances in deep learning. However, they tend to suffer from high sample complexity, which hinders their use in real-world domains. Alternatively, model-based reinforcement learning promises to reduce sample complexity, but tends to require careful tuning and to date have succeeded mainly in restrictive domains where simple models are sufficient for learning. In this paper, we analyze the behavior of vanilla model-based reinforcement learning methods when deep neural networks are used to learn both the model and the policy, and show that the learned policy tends to exploit regions where insufficient data is available for the model to be learned, causing instability in training. To overcome this issue, we propose to use an ensemble of models to maintain the model uncertainty and regularize the learning process. We further show that the use of likelihood ratio derivatives yields much more stable learning than backpropagation through time. Altogether, our approach Model-Ensemble Trust-Region Policy Optimization (ME-TRPO) significantly reduces the sample complexity compared to model-free deep RL methods on challenging continuous control benchmark tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑