基于高回放比与正则化的高效稀疏奖励目标条件强化学习
Efficient Sparse-Reward Goal-Conditioned Reinforcement Learning with a High Replay Ratio and Regularization
AI总结：
本文将高回放比正则化方法REDQ结合事后经验回放与目标Q值界定，扩展到稀疏奖励目标条件任务，并通过组件简化显著提升样本效率。
摘要：
具有高回放比（RR）和正则化的强化学习（RL）方法因其卓越的样本效率而受到关注。然而，这些方法主要是为稠密奖励任务开发的。本文旨在将这些RL方法扩展到稀疏奖励的目标条件任务。我们采用Randomized Ensemble Double Q-learning（REDQ，随机集成双Q学习）（Chen等人，2021），这是一种具有高RR和正则化的RL方法。为了将REDQ应用于稀疏奖励目标条件任务，我们对其进行了以下修改：（i）使用事后经验回放；（ii）对目标Q值进行界定。我们在Robotics（Plappert等人，2018）的12个稀疏奖励目标条件任务上评估了经过这些修改的REDQ，并表明其样本效率比此前最先进（SoTA）的RL方法提高约2倍。此外，我们重新审视了REDQ特定组件的必要性，并通过移除不必要组件对其进行简化。经过我们修改的简化版REDQ在Robotics的4个Fetch任务中实现了比SoTA方法约8倍的样本效率。
英文摘要：
Reinforcement learning (RL) methods with a high replay ratio (RR) and regularization have gained interest due to their superior sample efficiency. However, these methods have mainly been developed for dense-reward tasks. In this paper, we aim to extend these RL methods to sparse-reward goal-conditioned tasks. We use Randomized Ensemble Double Q-learning (REDQ) (Chen et al., 2021), an RL method with a high RR and regularization. To apply REDQ to sparse-reward goal-conditioned tasks, we make the following modifications to it: (i) using hindsight experience replay and (ii) bounding target Q-values. We evaluate REDQ with these modifications on 12 sparse-reward goal-conditioned tasks of Robotics (Plappert et al., 2018), and show that it achieves about $2 \times$ better sample efficiency than previous state-of-the-art (SoTA) RL methods. Furthermore, we reconsider the necessity of specific components of REDQ and simplify it by removing unnecessary ones. The simplified REDQ with our modifications achieves $\sim 8 \times$ better sample efficiency than the SoTA methods in 4 Fetch tasks of Robotics.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
