arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2004.14567cs.LGcs.ROstat.ML
收藏

改进强化学习的平面-空间状态嵌入

Plan-Space State Embeddings for Improved Reinforcement Learning

University of Southern California(南加州大学)

Max Pflueger, Gaurav S. Sukhatme

摘要:

机器人控制问题通常是用一个策略函数来构造的,该策略函数将状态值映射到控制值,但是在许多动态问题中,观察到的状态可能难以表征与有用的策略动作的关系。在本文中,我们提出了一种新的方法学习状态嵌入计划或其他形式的演示,使嵌入空间具有特定的几何关系的演示。我们提出了一种新的变分框架来学习这些嵌入,试图优化学习的嵌入空间中的轨迹线性。我们展示了如何将这些嵌入空间用作强化学习问题中机器人状态的增强。我们使用kinodynamic规划来生成一些示例环境的训练轨迹,然后为这些环境训练嵌入空间。我们的经验表明,在学习的嵌入空间中观察系统可以提高策略梯度强化学习算法的性能,特别是通过减少训练运行之间的方差。我们的技术仅限于演示数据可用的环境,但对如何收集数据没有限制。我们的嵌入技术提供了一种方法来转移领域知识从现有的技术,如规划和控制算法,到更灵活的策略学习算法,通过创建一个抽象的机器人状态表示有意义的几何形状。

英文摘要:

Robot control problems are often structured with a policy function that maps state values into control values, but in many dynamic problems the observed state can have a difficult to characterize relationship with useful policy actions. In this paper we present a new method for learning state embeddings from plans or other forms of demonstrations such that the embedding space has a specified geometric relationship with the demonstrations. We present a novel variational framework for learning these embeddings that attempts to optimize trajectory linearity in the learned embedding space. We show how these embedding spaces can then be used as an augmentation to the robot state in reinforcement learning problems. We use kinodynamic planning to generate training trajectories for some example environments, and then train embedding spaces for these environments. We show empirically that observing a system in the learned embedding space improves the performance of policy gradient reinforcement learning algorithms, particularly by reducing the variance between training runs. Our technique is limited to environments where demonstration data is available, but places no limits on how that data is collected. Our embedding technique provides a way to transfer domain knowledge from existing technologies such as planning and control algorithms, into more flexible policy learning algorithms, by creating an abstract representation of the robot state with meaningful geometry.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑