arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1907.11971cs.AI
收藏

面向工业环境的基于模型的强化学习

Towards Model-based Reinforcement Learning for Industry-near Environments

University of Agder(阿格德大学)

Per-Arne Andersen, Morten Goodwin, Ole-Christoffer Granmo

摘要:

深度强化学习在过去的几年里已经显示出在复杂的模拟环境中学习接近最优控制的巨大潜力,几乎没有可见的信息。Rainbow(Q-Learning)和PPO(Policy Optimization)在各种任务中表现出色,包括Atari 2600、MuJoCo和Roboschool测试套件。虽然这些算法从根本上是不同的,但两者都具有高方差、低样本效率和超参数敏感性,这在实践中使这些算法在行业中的关键操作中行不通。另一方面,基于模型的强化学习侧重于学习环境中状态之间的转换动态。如果这些环境动态是充分学习,基于模型的方法可能是最有效的方法,学习代理在环境中最佳地采取行动。基于模型的强化特性非常适合采样速度缓慢的现实环境和关键任务操作。在仓储行业,越来越多的人希望最大限度地缩短时间并最大限度地提高产量。目前,自主智能体的行为次优使用手工制作的政策的状态空间的重要部分。在本文中,我们提出了Dreaming变分自编码器v2(DVAE-2),这是一种基于基于模型的强化学习算法,可以提高样本效率,从而使低样本效率的算法在现实环境中更好地发挥作用。我们介绍了深仓库,一个模拟环境的行业近测试的自主智能体在基于网格的仓库。最后,我们说明了DVAE-2提高了样本效率的深度仓库相比,无模型的方法。

英文摘要:

Deep reinforcement learning has over the past few years shown great potential in learning near-optimal control in complex simulated environments with little visible information. Rainbow (Q-Learning) and PPO (Policy Optimisation) have shown outstanding performance in a variety of tasks, including Atari 2600, MuJoCo, and Roboschool test suite. While these algorithms are fundamentally different, both suffer from high variance, low sample efficiency, and hyperparameter sensitivity that in practice, make these algorithms a no-go for critical operations in the industry. On the other hand, model-based reinforcement learning focuses on learning the transition dynamics between states in an environment. If these environment dynamics are adequately learned, a model-based approach is perhaps the most sample efficient method for learning agents to act in an environment optimally. The traits of model-based reinforcement are ideal for real-world environments where sampling is slow and for mission-critical operations. In the warehouse industry, there is an increasing motivation to minimise time and to maximise production. Currently, autonomous agents act suboptimally using handcrafted policies for significant portions of the state-space. In this paper, we present The Dreaming Variational Autoencoder v2 (DVAE-2), a model-based reinforcement learning algorithm that increases sample efficiency, hence enable algorithms with low sample efficiency function better in real-world environments. We introduce Deep Warehouse, a simulated environment for industry-near testing of autonomous agents in grid-based warehouses. Finally, we illustrate that DVAE-2 improves the sample efficiency for the Deep Warehouse compared to model-free methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑