RL-Scope：深度强化学习工作负载的跨栈分析
RL-Scope: Cross-Stack Profiling for Deep Reinforcement Learning Workloads
摘要：
深度强化学习（RL）在机器人、数据中心管理和其他应用方面取得了突破性的进展。不幸的是，人们对RL工作负载中的系统级瓶颈知之甚少;我们观察到RL工作负载中的基本结构差异，这使得它们本质上比监督学习（SL）更少受GPU约束。为了解释RL工作负载中的训练时间，我们提出了RL-Scope，这是一个跨堆栈分析器，它将低级CPU/GPU资源使用情况限制在高级算法操作范围内，并通过纠正分析开销提供准确的见解。使用RL-Scope，我们调查了其主要维度的RL工作负载，包括ML后端，RL算法和模拟器。对于ML后端，我们解释了等效PyTorch和TensorFlow算法实现之间运行时的$2.3\times$差异，并确定了源于过度抽象算法实现的瓶颈。对于RL算法和模拟器，我们表明，对政策的算法至少$3.5\times$更多的模拟约束比关闭政策的算法。最后，我们分析了一个扩展的工作负载，并证明了常用工具报告的GPU利用率指标大大提高了GPU的使用率，而RL-Scope报告的是真正的GPU绑定时间。RL-Scope是一个开源工具，可在https://github.com/UofT-EcoSystem/rlscope上获得。
英文摘要：
Deep reinforcement learning (RL) has made groundbreaking advancements in robotics, data center management and other applications. Unfortunately, system-level bottlenecks in RL workloads are poorly understood; we observe fundamental structural differences in RL workloads that make them inherently less GPU-bound than supervised learning (SL). To explain where training time is spent in RL workloads, we propose RL-Scope, a cross-stack profiler that scopes low-level CPU/GPU resource usage to high-level algorithmic operations, and provides accurate insights by correcting for profiling overhead. Using RL-Scope, we survey RL workloads across its major dimensions including ML backend, RL algorithm, and simulator. For ML backends, we explain a $2.3\times$ difference in runtime between equivalent PyTorch and TensorFlow algorithm implementations, and identify a bottleneck rooted in overly abstracted algorithm implementations. For RL algorithms and simulators, we show that on-policy algorithms are at least $3.5\times$ more simulation-bound than off-policy algorithms. Finally, we profile a scale-up workload and demonstrate that GPU utilization metrics reported by commonly used tools dramatically inflate GPU usage, whereas RL-Scope reports true GPU-bound time. RL-Scope is an open-source tool available at https://github.com/UofT-EcoSystem/rlscope .
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
