arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1906.02771cs.LGcs.AIstat.ML
收藏

用规范化流策略改进软演员-评论的探索

Improving Exploration in Soft-Actor-Critic with Normalizing Flows Policies

McGill University(麦吉尔大学);Mila(米拉研究所)

Patrick Nadeem Ward, Ariella Smofsky, Avishek Joey Bose

摘要:

已知用于连续动作空间的深度强化学习(DRL)算法对于超参数是脆弱的,并且\cut{being}样本效率低下。Soft Actor Critic(SAC)在最大熵RL框架内提出了一种离线深度演员评论算法,该算法提供了更大的稳定性和经验增益。策略分布的选择,一个因子高斯,是由\cut{chosen due}它的容易重新参数化,而不是它的建模能力。我们在SAC框架内引入了规范化流策略,该策略比简单的因子高斯算法更具表达性。\cut{We also present a series of stabilization tricks that enable effective training of these policies in the RL setting.}我们在连续网格世界任务上的经验表明,我们的方法增加了稳定性,更适合于稀疏奖励设置中的困难探索。

英文摘要:

Deep Reinforcement Learning (DRL) algorithms for continuous action spaces are known to be brittle toward hyperparameters as well as \cut{being}sample inefficient. Soft Actor Critic (SAC) proposes an off-policy deep actor critic algorithm within the maximum entropy RL framework which offers greater stability and empirical gains. The choice of policy distribution, a factored Gaussian, is motivated by \cut{chosen due}its easy re-parametrization rather than its modeling power. We introduce Normalizing Flow policies within the SAC framework that learn more expressive classes of policies than simple factored Gaussians. \cut{We also present a series of stabilization tricks that enable effective training of these policies in the RL setting.}We show empirically on continuous grid world tasks that our approach increases stability and is better suited to difficult exploration in sparse reward settings.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑