D-STACK：通过GPU的高效复用与时空调度实现高吞吐量DNN推理
D-STACK: High Throughput DNN Inference by Effective Multiplexing and Spatio-Temporal Scheduling of GPUs
AI总结：
针对DNN推理中GPU利用率不足及空间复用下资源配置、批处理与调度难题，本文提出D-STACK时空调度框架，结合并行性估计模型与轻量级批处理优化，在满足SLO的同时显著提升GPU利用率和推理吞吐量。
摘要：
使用深度神经网络（DNN）进行实时、低延迟推理需要GPU等硬件加速器。然而，由于DNN能够利用的并行性存在固有局限，它们往往无法充分利用当今高端加速器的容量。尽管GPU的空间复用能够提高GPU利用率和推理吞吐量，但仍存在若干挑战。通过性能分析为每个DNN确定合适的GPU百分比以实现GPU规模的合理配置、确定最优请求批处理大小以在满足应用特定截止期限和服务水平目标（SLO）的同时平衡吞吐量提升，以及通过适当调度DNN来最大化吞吐量，仍然是重大挑战。本文提出了一种动态且公平的时空调度器（D-STACK），使多个DNN能够在GPU中并发运行。为帮助分配适当的GPU百分比（我们称之为“Knee”），我们开发并验证了一个模型，用于估计每个DNN能够利用的并行性。我们还开发了一种轻量级优化公式，用于为在D-STACK上运行的每个DNN寻找高效的批处理大小。我们将这些优化与时空调度器结合起来，提供了一个整体推理框架。我们展示了其在满足应用SLO的同时提供高吞吐量的能力。我们将D-STACK与一个能够为每个DNN内核分配合适GPU百分比的理想调度器进行比较。与该理想调度器相比，D-STACK实现了超过90%的吞吐量和GPU利用率。我们还使用流行的DNN模型，将D-STACK与其他GPU复用和调度方法（例如NVIDIA Triton、Clipper、Nexus）进行比较。我们在复用多个流行DNN模型的受控实验中，实现了最高1.6倍的GPU利用率提升和最高4倍的推理吞吐量提升。
英文摘要：
Hardware accelerators such as GPUs are required for real-time, low-latency inference with Deep Neural Networks (DNN). However, due to the inherent limits to the parallelism they can exploit, DNNs often under-utilize the capacity of today's high-end accelerators. Although spatial multiplexing of the GPU, leads to higher GPU utilization and higher inference throughput, there remain a number of challenges. Finding the GPU percentage for right-sizing the GPU for each DNN through profiling, determining an optimal batching of requests to balance throughput improvement while meeting application-specific deadlines and service level objectives (SLOs), and maximizing throughput by appropriately scheduling DNNs are still significant challenges. This paper introduces a dynamic and fair spatio-temporal scheduler (D-STACK) that enables multiple DNNs to run in the GPU concurrently. To help allocate the appropriate GPU percentage (we call it the "Knee"), we develop and validate a model that estimates the parallelism each DNN can utilize. We also develop a lightweight optimization formulation to find an efficient batch size for each DNN operating with D-STACK. We bring together our optimizations and our spatio-temporal scheduler to provide a holistic inference framework. We demonstrate its ability to provide high throughput while meeting application SLOs. We compare D-STACK with an ideal scheduler that can allocate the right GPU percentage for every DNN kernel. D-STACK gets higher than 90 percent throughput and GPU utilization compared to the ideal scheduler. We also compare D-STACK with other GPU multiplexing and scheduling methods (e.g., NVIDIA Triton, Clipper, Nexus), using popular DNN models. Our controlled experiments with multiplexing several popular DNN models achieve up to 1.6X improvement in GPU utilization and up to 4X improvement in inference throughput.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
