WarpDrive:基于GPU的极快端到端深度多智能体强化学习
WarpDrive: Extremely Fast End-to-End Deep Multi-Agent Reinforcement Learning on a GPU
摘要:
深度强化学习(RL)是一种在复杂环境中训练决策模型的强大框架。然而,RL可能很慢,因为它需要与环境模拟进行重复交互。特别是,当在复杂的环境中使用RL时,存在关键的系统工程瓶颈,这些环境具有高维状态,观察或动作空间的多个代理。WarpDrive是一个灵活,轻量级,易于使用的开源RL框架,它在单个GPU(图形处理单元)上实现了端到端的深度多代理RL,构建在PyCUDA和PyTorch上。与混合CPU模拟和GPU模型的常见实现相比,WarpDrive使用GPU的极端并行化能力实现了数量级更快的RL。我们的设计运行模拟和代理在每个模拟并行。它消除了CPU和GPU之间的数据复制。它还使用GPU上的单个模拟数据存储,可以安全地就地更新。WarpDrive提供了一个轻量级的Python接口和灵活的环境包装器,易于使用和扩展。总之,这使得用户可以轻松地运行数千个并发的多智能体模拟和训练的经验非常大的批次。通过大量的实验,我们验证了WarpDrive提供高吞吐量和几乎线性扩展到许多代理和并行环境。例如,WarpDrive在一个基准Tag模拟中,在2000个环境和1000个代理的情况下,每秒产生290万个环境步数(与CPU实现相比,吞吐量至少高出100倍)。因此,WarpDrive是一个快速且可扩展的多代理RL平台,可以显着加速研究和开发。
英文摘要:
Deep reinforcement learning (RL) is a powerful framework to train decision-making models in complex environments. However, RL can be slow as it requires repeated interaction with a simulation of the environment. In particular, there are key system engineering bottlenecks when using RL in complex environments that feature multiple agents with high-dimensional state, observation, or action spaces. We present WarpDrive, a flexible, lightweight, and easy-to-use open-source RL framework that implements end-to-end deep multi-agent RL on a single GPU (Graphics Processing Unit), built on PyCUDA and PyTorch. Using the extreme parallelization capability of GPUs, WarpDrive enables orders-of-magnitude faster RL compared to common implementations that blend CPU simulations and GPU models. Our design runs simulations and the agents in each simulation in parallel. It eliminates data copying between CPU and GPU. It also uses a single simulation data store on the GPU that is safely updated in-place. WarpDrive provides a lightweight Python interface and flexible environment wrappers that are easy to use and extend. Together, this allows the user to easily run thousands of concurrent multi-agent simulations and train on extremely large batches of experience. Through extensive experiments, we verify that WarpDrive provides high-throughput and scales almost linearly to many agents and parallel environments. For example, WarpDrive yields 2.9 million environment steps/second with 2000 environments and 1000 agents (at least 100x higher throughput compared to a CPU implementation) in a benchmark Tag simulation. As such, WarpDrive is a fast and extensible multi-agent RL platform to significantly accelerate research and development.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。