arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2005.12108cs.LGstat.ML
收藏

梯度逼近的强化学习

Gradient Monitored Reinforcement Learning

South Westphalia University of Applied Sciences(西南威斯特伐利亚应用科学大学);University of Duisburg-Essen(杜伊斯堡-埃森大学)

Mohammed Sharafath Abdul Hameed, Gavneet Singh Chadha, Andreas Schwung, Steven X. Ding

摘要:

本文提出了一种新的神经网络训练方法,用于深度强化学习中更快的收敛和更好的泛化能力。特别是,我们专注于通过系统地降低梯度的方差来增强强化学习算法的训练和评估性能,从而提供更有针对性的学习过程。梯度监控(Gradient Monitoring,GM)方法是一种基于训练过程本身的动态发展和反馈来引导神经网络权值参数学习的方法。我们提出了不同的变种的GM方法,已被证明可以提高模型的基本性能。所提出的变体之一,具有梯度监测的动量(M-WGM),允许基于某些学习参数连续调整网络中反向传播梯度的量子。我们进一步增强了自适应动量梯度监测(AM-WGM)方法,该方法允许根据收集的奖励反馈在特定权重的集中学习与更分散的学习之间进行自动调整。作为一个副产品,它还允许在训练过程中自动导出所需的深度网络大小,因为算法会自动冻结训练的权重。该方法被应用到两个离散的(多机器人协调问题和雅达利游戏)和一个连续的控制任务(MuJoCo)分别使用优势Actor-Critic(A2 C)和近端策略优化(PPO)。所获得的结果特别强调的泛化能力方面的方法的适用性和性能改进。

英文摘要:

This paper presents a novel neural network training approach for faster convergence and better generalization abilities in deep reinforcement learning. Particularly, we focus on the enhancement of training and evaluation performance in reinforcement learning algorithms by systematically reducing gradient's variance and thereby providing a more targeted learning process. The proposed method which we term as Gradient Monitoring(GM), is an approach to steer the learning in the weight parameters of a neural network based on the dynamic development and feedback from the training process itself. We propose different variants of the GM methodology which have been proven to increase the underlying performance of the model. The one of the proposed variant, Momentum with Gradient Monitoring (M-WGM), allows for a continuous adjustment of the quantum of back-propagated gradients in the network based on certain learning parameters. We further enhance the method with Adaptive Momentum with Gradient Monitoring (AM-WGM) method which allows for automatic adjustment between focused learning of certain weights versus a more dispersed learning depending on the feedback from the rewards collected. As a by-product, it also allows for automatic derivation of the required deep network sizes during training as the algorithm automatically freezes trained weights. The approach is applied to two discrete (Multi-Robot Co-ordination problem and Atari games) and one continuous control task (MuJoCo) using Advantage Actor-Critic (A2C) and Proximal Policy Optimization (PPO) respectively. The results obtained particularly underline the applicability and performance improvements of the methods in terms of generalization capability.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑