arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2004.07804cs.LGcs.AIcs.ROstat.ML
收藏

基于基于模型的强化学习的博弈论框架

A Game Theoretic Framework for Model Based Reinforcement Learning

Google Brain(谷歌大脑);University of Washington(华盛顿大学)

Aravind Rajeswaran, Igor Mordatch, Vikash Kumar

摘要:

基于模型的强化学习(MBRL)由于其潜在的样本效率和合并非策略数据的能力而最近获得了极大的兴趣。然而,使用丰富的函数逼近器设计稳定和高效的MBRL算法仍然具有挑战性。为了帮助暴露MBRL的实际挑战,并从抽象的角度简化算法设计,我们开发了一个新的框架,将MBRL转换为一个游戏:(1)策略参与者,试图在学习模型下最大化奖励;(2)模型参与者,试图拟合策略参与者收集的真实数据。对于算法开发,我们构建了一个Stackelberg游戏之间的两个球员,并表明它可以解决近似的双层优化。这就产生了两个自然家庭的算法MBRL的基础上,球员被选为领导者在Stackelberg游戏。它们一起封装、统一和概括了许多以前的MBRL算法。此外,我们的框架是一致的,并提供了一个明确的基础,从以前的作品在实践中是重要的。最后,通过实验,我们验证了我们提出的算法是高度采样效率,匹配无模型策略梯度的渐近性能,并优雅地扩展到高维任务,如灵巧手操作。其他详细信息和代码可以从项目页面https://sites.google.com/view/mbrl-game获得

英文摘要:

Model-based reinforcement learning (MBRL) has recently gained immense interest due to its potential for sample efficiency and ability to incorporate off-policy data. However, designing stable and efficient MBRL algorithms using rich function approximators have remained challenging. To help expose the practical challenges in MBRL and simplify algorithm design from the lens of abstraction, we develop a new framework that casts MBRL as a game between: (1) a policy player, which attempts to maximize rewards under the learned model; (2) a model player, which attempts to fit the real-world data collected by the policy player. For algorithm development, we construct a Stackelberg game between the two players, and show that it can be solved with approximate bi-level optimization. This gives rise to two natural families of algorithms for MBRL based on which player is chosen as the leader in the Stackelberg game. Together, they encapsulate, unify, and generalize many previous MBRL algorithms. Furthermore, our framework is consistent with and provides a clear basis for heuristics known to be important in practice from prior works. Finally, through experiments we validate that our proposed algorithms are highly sample efficient, match the asymptotic performance of model-free policy gradient, and scale gracefully to high-dimensional tasks like dexterous hand manipulation. Additional details and code can be obtained from the project page at https://sites.google.com/view/mbrl-game

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑