Tsallis熵正则化下Bootstrapped Q集成深度强化学习的有效探索
Effective Exploration for Deep Reinforcement Learning via Bootstrapped Q-Ensembles under Tsallis Entropy Regularization
摘要：
近年来，深度强化学习（Deep Reinforcement Learning，DRL）在解决许多困难的大规模强化学习问题上取得了显著的成功。然而，有效学习所需的高样本成本往往使DRL在资源有限的应用中负担不起。为了提高样本效率和学习性能，本文提出了一种新的DRL算法，该算法无缝集成了熵诱导和引导诱导技术，以有效和深入地探索学习环境。具体来说，Tsallis熵正则化的一般形式将被用来驱动熵诱导的探索的基础上有效的近似最佳的行动选择政策。不同于现有的许多作品，依赖于行动抖动策略的探索，我们的算法是有效的，探索明确的探索价值的行动。同时，通过采用不同Tsallis熵正则化下的Q-网络的系综，可以进一步增强系综的多样性，以实现有效的引导诱导探索。在Atari游戏任务上的实验清楚地表明，与最近提出的探索方法（包括Bootstrapped Deep Q-Network和UCB Q-Ensemble）相比，我们的新算法可以实现更高效和有效的DRL探索。
英文摘要：
Recently deep reinforcement learning (DRL) has achieved outstanding success on solving many difficult and large-scale RL problems. However the high sample cost required for effective learning often makes DRL unaffordable in resource-limited applications. With the aim of improving sample efficiency and learning performance, we will develop a new DRL algorithm in this paper that seamless integrates entropy-induced and bootstrap-induced techniques for efficient and deep exploration of the learning environment. Specifically, a general form of Tsallis entropy regularizer will be utilized to drive entropy-induced exploration based on efficient approximation of optimal action-selection policies. Different from many existing works that rely on action dithering strategies for exploration, our algorithm is efficient in exploring actions with clear exploration value. Meanwhile, by employing an ensemble of Q-networks under varied Tsallis entropy regularization, the diversity of the ensemble can be further enhanced to enable effective bootstrap-induced exploration. Experiments on Atari game playing tasks clearly demonstrate that our new algorithm can achieve more efficient and effective exploration for DRL, in comparison to recently proposed exploration methods including Bootstrapped Deep Q-Network and UCB Q-Ensemble.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
