利用稀疏动态值估计增强场景特异性
Enhanced Scene Specificity with Sparse Dynamic Value Estimation
摘要：
多场景强化学习涉及从同一任务跨多个场景/级别训练RL代理，并且已经成为许多泛化应用的必要条件。然而，包含多个场景导致策略梯度计算的样本方差增加，通常导致直接应用传统方法（例如PPO、A3 C）的次优性能。一种用于方差减小的策略是将每个场景视为不同的马尔可夫决策过程（MDP），并且学习依赖于状态和MDP（M）两者的联合价值函数。然而，这并不是微不足道的，因为代理通常不知道多场景RL中的训练/测试时间的底层级别。最近，Singh等人[1]试图通过提出一种动态值估计方法来解决这个问题，该方法将真实的联合价值函数分布建模为高斯混合模型（GMM）。在本文中，我们认为，真正的场景特定的价值函数和预测的动态估计之间的错误可以进一步减少逐步执行稀疏集群分配一旦代理探索了大部分的状态空间。由此产生的代理不仅在一系列OpenAI ProcGen环境中显示出最终奖励分数的显着改善，而且在完成游戏关卡时还表现出更高的导航效率。
英文摘要：
Multi-scene reinforcement learning involves training the RL agent across multiple scenes / levels from the same task, and has become essential for many generalization applications. However, the inclusion of multiple scenes leads to an increase in sample variance for policy gradient computations, often resulting in suboptimal performance with the direct application of traditional methods (e.g. PPO, A3C). One strategy for variance reduction is to consider each scene as a distinct Markov decision process (MDP) and learn a joint value function dependent on both state (s) and MDP (M). However, this is non-trivial as the agent is usually unaware of the underlying level at train / test times in multi-scene RL. Recently, Singh et al. [1] tried to address this by proposing a dynamic value estimation approach that models the true joint value function distribution as a Gaussian mixture model (GMM). In this paper, we argue that the error between the true scene-specific value function and the predicted dynamic estimate can be further reduced by progressively enforcing sparse cluster assignments once the agent has explored most of the state space. The resulting agents not only show significant improvements in the final reward score across a range of OpenAI ProcGen environments, but also exhibit increased navigation efficiency while completing a game level.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
