基于多任务神经进化的长短期强化学习
Multitask Neuroevolution for Reinforcement Learning with Long and Short Episodes
摘要:
研究表明演化策略(ES)是深度神经网络的强化学习(RL)的一种有前途的方法。然而,高样本复杂度的问题仍然存在于ES在长期的深度RL应用中。本文是第一个通过一种新的神经进化多任务(NuEMT)算法来解决当今方法的缺点,该算法旨在将信息从一组辅助任务(短情节长度)转移到目标(全长)RL任务。从目标中提取的辅助任务允许代理在较短的时间范围内更新和快速评估策略。然后,进化的技能被转移到指导更长和更难的任务,以实现最佳策略。我们证明了NuEMT算法实现了数据高效的进化RL,减少了昂贵的代理-环境交互数据需求。我们的关键算法的贡献,在这种情况下,是第一次介绍,一个多任务的技能转移机制的基础上的统计重要性采样技术。此外,自适应资源分配策略被用来分配计算资源的辅助任务的基础上收集的有用性。在OpenAI Gym的一系列连续控制任务上进行的实验证实,与最近的ES基线相比,我们提出的算法是有效的。
英文摘要:
Studies have shown evolution strategies (ES) to be a promising approach for reinforcement learning (RL) with deep neural networks. However, the issue of high sample complexity persists in applications of ES to deep RL over long horizons. This paper is the first to address the shortcoming of today's methods via a novel neuroevolutionary multitasking (NuEMT) algorithm, designed to transfer information from a set of auxiliary tasks (of short episode length) to the target (full length) RL task at hand. The auxiliary tasks, extracted from the target, allow an agent to update and quickly evaluate policies on shorter time horizons. The evolved skills are then transferred to guide the longer and harder task towards an optimal policy. We demonstrate that the NuEMT algorithm achieves data-efficient evolutionary RL, reducing expensive agent-environment interaction data requirements. Our key algorithmic contribution in this setting is to introduce, for the first time, a multitask skills transfer mechanism based on the statistical importance sampling technique. In addition, an adaptive resource allocation strategy is utilized to assign computational resources to auxiliary tasks based on their gleaned usefulness. Experiments on a range of continuous control tasks from the OpenAI Gym confirm that our proposed algorithm is efficient compared to recent ES baselines.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。