稳定基线中深度Q学习的Q值演化
Evolution of Q Values for Deep Q Learning in Stable Baselines
摘要：
我们研究了在稳定基线库中实现深度Q学习（DQL）的Q值的演变。稳定基线结合了最新的强化学习技术，并在许多游戏环境中实现了超人的性能。然而，对于一些简单的非游戏环境，稳定基线中的DQL可能很难找到正确的操作。在本文中，我们的目标是了解这种次优行为可能发生的环境类型，并研究相应的演化的Q值为个别国家。我们比较了一个智能灯光环境（性能很差）和AI Gym FrozenLake环境（性能很好）。我们观察到DQL与FrozenLight的斗争，因为动作是可逆的，因此给定状态下的Q值比FrozenLake更接近。然后，我们调查的Q值使用最近的分解技术的Achiam等人的演变。我们观察到，对于pixicLight，函数近似误差和状态之间的复杂关系导致了一些Q值远离最优的情况。
英文摘要：
We investigate the evolution of the Q values for the implementation of Deep Q Learning (DQL) in the Stable Baselines library. Stable Baselines incorporates the latest Reinforcement Learning techniques and achieves superhuman performance in many game environments. However, for some simple non-game environments, the DQL in Stable Baselines can struggle to find the correct actions. In this paper we aim to understand the types of environment where this suboptimal behavior can happen, and also investigate the corresponding evolution of the Q values for individual states. We compare a smart TrafficLight environment (where performance is poor) with the AI Gym FrozenLake environment (where performance is perfect). We observe that DQL struggles with TrafficLight because actions are reversible and hence the Q values in a given state are closer than in FrozenLake. We then investigate the evolution of the Q values using a recent decomposition technique of Achiam et al.. We observe that for TrafficLight, the function approximation error and the complex relationships between the states lead to a situation where some Q values meander far from optimal.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
