使深度Q学习方法对时间离散化具有鲁棒性
Making Deep Q-learning methods robust to time discretization
摘要:
尽管取得了显著的成功,但深度强化学习(DRL)对超参数化、实现细节或小的环境变化并不鲁棒(Henderson et al. 2017,Zhang et al. 2018)。克服这种敏感性是使DRL适用于现实世界问题的关键。在本文中,我们确定在近连续时间环境中对时间离散化的敏感性是一个关键因素;这包括,例如,改变每秒帧数或控制器的动作频率。从经验上讲,我们发现基于Q学习的方法,如深度Q学习(Mnih et al.,2015)和深度确定性策略梯度(Lillicrap等人,2015年,以小的时间步长崩溃。形式上,我们证明了Q学习在连续时间中不存在。我们详细介绍了一种原则性的方法来构建一个非策略RL算法,该算法在广泛的时间离散化范围内产生类似的性能,并根据经验证实了这种鲁棒性。
英文摘要:
Despite remarkable successes, Deep Reinforcement Learning (DRL) is not robust to hyperparameterization, implementation details, or small environment changes (Henderson et al. 2017, Zhang et al. 2018). Overcoming such sensitivity is key to making DRL applicable to real world problems. In this paper, we identify sensitivity to time discretization in near continuous-time environments as a critical factor; this covers, e.g., changing the number of frames per second, or the action frequency of the controller. Empirically, we find that Q-learning-based approaches such as Deep Q- learning (Mnih et al., 2015) and Deep Deterministic Policy Gradient (Lillicrap et al., 2015) collapse with small time steps. Formally, we prove that Q-learning does not exist in continuous time. We detail a principled way to build an off-policy RL algorithm that yields similar performances over a wide range of time discretizations, and confirm this robustness empirically.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。