arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1911.07643cs.LGstat.ML
收藏

用于深度强化学习的影响感知存储结构

Influence-aware Memory Architectures for Deep Reinforcement Learning

Delft University of Technology(代尔夫特理工大学)

Miguel Suau, Jinke He, Elena Congeduti, Rolf A. N. Starre, Aleksander Czechowski, Frans A. Oliehoek

摘要:

由于感知的局限性,智能体可能对环境状态的信息太少,无法采取最佳行动。在这种情况下,重要的是跟踪观察历史以发现隐藏的状态。最近的深度强化学习方法使用递归神经网络(RNN)来记忆过去的观察结果。然而,这些模型是昂贵的训练和收敛困难,特别是当处理高维输入空间。在本文中,我们提出了影响感知记忆(IAM),一个理论启发的内存架构,试图通过限制输入的递归层的变量,影响隐藏的状态信息,以减轻训练困难。此外,与标准RNN相反,用于估计Q值的每一条信息都不可避免地反馈到网络中进行下一次预测,我们的模型允许信息流动,而不必存储在RNN的内部存储器中。结果表明,通过让递归层专注于一小部分观测变量,同时用前馈神经网络处理其余信息,我们可以在训练速度和策略性能方面优于标准递归架构。这种方法还减少了运行时间,并获得比堆叠多个观测以去除部分可观测性的方法更好的分数。

英文摘要:

Due to its perceptual limitations, an agent may have too little information about the state of the environment to act optimally. In such cases, it is important to keep track of the observation history to uncover hidden state. Recent deep reinforcement learning methods use recurrent neural networks (RNN) to memorize past observations. However, these models are expensive to train and have convergence difficulties, especially when dealing with high dimensional input spaces. In this paper, we propose influence-aware memory (IAM), a theoretically inspired memory architecture that tries to alleviate the training difficulties by restricting the input of the recurrent layers to those variables that influence the hidden state information. Moreover, as opposed to standard RNNs, in which every piece of information used for estimating Q values is inevitably fed back into the network for the next prediction, our model allows information to flow without being necessarily stored in the RNN's internal memory. Results indicate that, by letting the recurrent layers focus on a small fraction of the observation variables while processing the rest of the information with a feedforward neural network, we can outperform standard recurrent architectures both in training speed and policy performance. This approach also reduces runtime and obtains better scores than methods that stack multiple observations to remove partial observability.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑