arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1910.01741cs.LGcs.AIcs.ROstat.ML
收藏

基于图像的无模型强化学习中样本效率的改进

Improving Sample Efficiency in Model-Free Reinforcement Learning from Images

New York University(纽约大学);Facebook AI Research(Facebook人工智能研究院);McGill University(麦吉尔大学);MILA(蒙特利尔学习算法研究所)

Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, Rob Fergus

摘要:

用无模型强化学习(RL)直接从高维图像训练智能体解决控制任务已经被证明是困难的。一个有前途的方法是学习一个潜在的代表与控制策略。然而,使用稀缺的奖励信号来拟合高容量编码器是采样效率低的,并且导致性能差。先前的工作已经表明,辅助损失,如图像重建,可以帮助有效的表示学习。然而,将重建损失并入离策略学习算法中通常会导致训练不稳定。我们探讨的根本原因,并确定变分自编码器,以前的调查,作为分歧的原因。根据这些发现,我们提出了有效的技术,以提高训练的稳定性。这导致一种简单的方法,能够在MuJoCo控制任务上匹配最先进的无模型和基于模型的算法。此外,我们的方法表现出对观测噪声的鲁棒性,超过了现有的方法。代码、结果和视频可在https://sites.google.com/view/sac-ae/home上匿名获取。

英文摘要:

Training an agent to solve control tasks directly from high-dimensional images with model-free reinforcement learning (RL) has proven difficult. A promising approach is to learn a latent representation together with the control policy. However, fitting a high-capacity encoder using a scarce reward signal is sample inefficient and leads to poor performance. Prior work has shown that auxiliary losses, such as image reconstruction, can aid efficient representation learning. However, incorporating reconstruction loss into an off-policy learning algorithm often leads to training instability. We explore the underlying reasons and identify variational autoencoders, used by previous investigations, as the cause of the divergence. Following these findings, we propose effective techniques to improve training stability. This results in a simple approach capable of matching state-of-the-art model-free and model-based algorithms on MuJoCo control tasks. Furthermore, our approach demonstrates robustness to observational noise, surpassing existing approaches in this setting. Code, results, and videos are anonymously available at https://sites.google.com/view/sac-ae/home.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑