arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1807.08452cs.LGcs.AIstat.ML
收藏

使用策略梯度学习来玩乒乓球

Learning to Play Pong using Policy Gradient Learning

Universiti Teknologi Brunei(文莱理工大学)

Somnuk Phon-Amnuaisuk

摘要:

强化学习(RL)中的活动围绕着学习马尔可夫决策过程(MDP)模型,特别是以下参数:状态值V;状态-动作值Q;以及策略pi。这些参数通常以数组的形式实现。放大问题意味着放大阵列的大小,这将很快导致计算瓶颈。为了解决这个问题,RL问题通常被制定为使用手工制作的输入功能来限制数组的大小来学习特定任务。在本报告中,我们讨论了另一种端到端深度强化学习(DRL)方法,其中DRL试图学习一般任务表示,在我们的上下文中,这是指学习从一系列屏幕快照中玩Pong游戏,而没有游戏特定的手工制作功能。我们应用人工神经网络(ANN)近似的RL模型的政策。策略网络通过策略梯度(PG)方法从一系列帧学习玩Pong游戏,除了像素信息和分数之外没有任何额外的语义。与传统的表格RL方法相比,数组中的内容具有明确的解释,如V或Q,从策略网络的权重对知识内容的解释更具迷惑性。在这项工作中,我们实验了各种深度ANN架构,即,前馈ANN(FFNN)、卷积ANN(CNN)和异步优势演员-评论家(A3 C)。我们还研究了隐藏节点的激活和输入和隐藏层之间的权重,之前和之后的DRL已经成功地学会了玩乒乓球游戏。深入了解内部学习机制和未来的研究方向进行了讨论。

英文摘要:

Activities in reinforcement learning (RL) revolve around learning the Markov decision process (MDP) model, in particular, the following parameters: state values, V; state-action values, Q; and policy, pi. These parameters are commonly implemented as an array. Scaling up the problem means scaling up the size of the array and this will quickly lead to a computational bottleneck. To get around this, the RL problem is commonly formulated to learn a specific task using hand-crafted input features to curb the size of the array. In this report, we discuss an alternative end-to-end Deep Reinforcement Learning (DRL) approach where the DRL attempts to learn general task representations which in our context refers to learning to play the Pong game from a sequence of screen snapshots without game-specific hand-crafted features. We apply artificial neural networks (ANN) to approximate a policy of the RL model. The policy network, via Policy Gradients (PG) method, learns to play the Pong game from a sequence of frames without any extra semantics apart from the pixel information and the score. In contrast to the traditional tabular RL approach where the contents in the array have clear interpretations such as V or Q, the interpretation of knowledge content from the weights of the policy network is more illusive. In this work, we experiment with various Deep ANN architectures i.e., Feed forward ANN (FFNN), Convolution ANN (CNN) and Asynchronous Advantage Actor-Critic (A3C). We also examine the activation of hidden nodes and the weights between the input and the hidden layers, before and after the DRL has successfully learnt to play the Pong game. Insights into the internal learning mechanisms and future research directions are then discussed.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑