arXivDaily arXiv每日学术速递 周一至周五更新
看起来今天可以划水了
arXiv 2007.13134cs.ROcs.LG
收藏

使用强化学习和生成模型的数据高效可视化策略训练

Data-efficient visuomotor policy training using reinforcement learning and generative models

KTH Royal Institute of Technology（瑞典皇家理工学院）；Aalto University（阿尔托大学）

Ali Ghadirzadeh, Petra Poklukar, Ville Kyrki, Danica Kragic, Mårten Björkman

阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

摘要：

我们提出了一个数据高效的框架，解决可视化顺序决策问题，利用强化学习（RL）和潜变量生成模型的组合。我们的框架通过引入动作潜在变量来训练深度可视化策略，使得前馈策略搜索可以分为三个部分：（i）训练子策略，该子策略在给定系统状态的情况下输出关于动作潜在变量的分布，（ii）无监督地训练生成模型，该生成模型输出以潜在动作变量为条件的运动动作序列，以及（iii）以端到端的方式对深度可视化策略进行监督训练。我们的方法可以安全探索，并解决数据效率低下的问题，因为它利用了有关有效的电机动作序列的先验知识。此外，我们还提供了一组用于评估生成模型的措施，以便我们能够在物理机器人的实际训练之前预测RL策略训练的性能。我们定义了两个新的措施，去纠缠和本地线性评估潜在的代表性的质量，并补充他们与现有的措施，评估的学习分布。我们通过实验确定了不同的生成模型的特性，这些模型对机器人采摘任务的最终策略训练的性能影响最大。

英文摘要：

We present a data-efficient framework for solving visuomotor sequential decision-making problems which exploits the combination of reinforcement learning (RL) and latent variable generative models. Our framework trains deep visuomotor policies by introducing an action latent variable such that the feed-forward policy search can be divided into three parts: (i) training a sub-policy that outputs a distribution over the action latent variable given a state of the system, (ii) unsupervised training of a generative model that outputs a sequence of motor actions conditioned on the latent action variable, and (iii) supervised training of the deep visuomotor policy in an end-to-end fashion. Our approach enables safe exploration and alleviates the data-inefficiency problem as it exploits prior knowledge about valid sequences of motor actions. Moreover, we provide a set of measures for evaluation of generative models such that we are able to predict the performance of the RL policy training prior to the actual training on a physical robot. We define two novel measures of disentanglement and local linearity for assessing the quality of latent representations, and complement them with existing measures for assessment of the learned distribution. We experimentally determine the characteristics of different generative models that have the most influence on performance of the final policy training on a robotic picking task.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑