基于稀疏注意的单任务多场景强化学习动态值估计
Sparse Attention Guided Dynamic Value Estimation for Single-Task Multi-Scene Reinforcement Learning
摘要：
在具有来自同一任务的多个级别/场景的环境中训练深度强化学习代理，对于许多旨在实现从模拟到现实世界的泛化和域转移的应用来说已经变得至关重要。虽然这种策略有助于泛化，但使用多个场景会显著增加为策略梯度计算收集的样本的方差。当前方法有效地继续将该场景集合视为单个马尔可夫决策过程（MDP），并且因此学习场景通用价值函数V（s）。然而，我们认为多场景环境的样本方差最好通过将每个场景视为不同的MDP来最小化，然后学习依赖于状态s和MDP M的联合价值函数V（s，M）。我们进一步证明了多场景环境的真正联合价值函数遵循传统的基于CNN / LSTM的评论家网络无法捕获的多模态分布。为此，我们提出了一种动态值估计（DVE）技术，它通过稀疏的注意力机制在多个价值函数假设/模式近似的真正的联合价值函数。由此产生的代理不仅在一系列OpenAI ProcGen环境中显示出最终奖励分数的显着改善，而且还表现出增强的导航效率，并为无监督状态空间技能分解提供了一种隐式机制。
英文摘要：
Training deep reinforcement learning agents on environments with multiple levels / scenes from the same task, has become essential for many applications aiming to achieve generalization and domain transfer from simulation to the real world. While such a strategy is helpful with generalization, the use of multiple scenes significantly increases the variance of samples collected for policy gradient computations. Current methods, effectively continue to view this collection of scenes as a single Markov decision process (MDP), and thus learn a scene-generic value function V(s). However, we argue that the sample variance for a multi-scene environment is best minimized by treating each scene as a distinct MDP, and then learning a joint value function V(s,M) dependent on both state s and MDP M. We further demonstrate that the true joint value function for a multi-scene environment, follows a multi-modal distribution which is not captured by traditional CNN / LSTM based critic networks. To this end, we propose a dynamic value estimation (DVE) technique, which approximates the true joint value function through a sparse attention mechanism over multiple value function hypothesis / modes. The resulting agent not only shows significant improvements in the final reward score across a range of OpenAI ProcGen environments, but also exhibits enhanced navigation efficiency and provides an implicit mechanism for unsupervised state-space skill decomposition.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
