随机集成双Q学习:无需模型快速学习
Randomized Ensembled Double Q-Learning: Learning Fast Without a Model
摘要:
使用高的更新到数据(UTD)的比率,基于模型的方法最近取得了更高的样本效率比以前的无模型方法的连续行动DRL基准。在本文中,我们介绍了一个简单的无模型算法,随机集成双Q学习(REDQ),并表明其性能与MuJoCo基准测试的最先进的基于模型的算法一样好,如果不是更好的话。此外,REDQ可以使用比基于模型的方法更少的参数来实现该性能,并且具有更少的挂钟运行时间。REDQ有三个精心集成的成分,使其能够实现其高性能:(i)UTD比>> 1;(ii)Q函数的集合;(iii)来自集合的Q函数的随机子集上的目标最小化。通过精心设计的实验,我们提供了一个详细的分析REDQ和相关的无模型算法。据我们所知,REDQ是第一个成功的无模型DRL算法,用于连续作用空间,使用UTD比>> 1。
英文摘要:
Using a high Update-To-Data (UTD) ratio, model-based methods have recently achieved much higher sample efficiency than previous model-free methods for continuous-action DRL benchmarks. In this paper, we introduce a simple model-free algorithm, Randomized Ensembled Double Q-Learning (REDQ), and show that its performance is just as good as, if not better than, a state-of-the-art model-based algorithm for the MuJoCo benchmark. Moreover, REDQ can achieve this performance using fewer parameters than the model-based method, and with less wall-clock run time. REDQ has three carefully integrated ingredients which allow it to achieve its high performance: (i) a UTD ratio >> 1; (ii) an ensemble of Q functions; (iii) in-target minimization across a random subset of Q functions from the ensemble. Through carefully designed experiments, we provide a detailed analysis of REDQ and related model-free algorithms. To our knowledge, REDQ is the first successful model-free DRL algorithm for continuous-action spaces using a UTD ratio >> 1.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。