GPU的空间共享，用于自动调整DNN模型
Spatial Sharing of GPU for Autotuning DNN models
摘要：
GPU用于训练、推断和调整机器学习模型。然而，深度神经网络（DNN）在充分利用高性能GPU的能力方面差异很大。GPU的空间共享使得能够在GPU上复用多个DNN，并且可以提高GPU利用率，从而提高吞吐量并降低延迟。DNN模型只要有适量的GPU资源，仍然可以提供低推理延迟，就像将所有GPU专用于其推理任务一样。改进DNN推理的一种方法是调整DNN模型。自动调优框架基于经过训练的机器学习模型为特定目标设备找到最佳低级实现，从而减少DNN的推理延迟并提高推理吞吐量。我们观察到一个相互依赖的调整模型和它的推理延迟。使用特定GPU资源调整的DNN模型在使用接近相同数量的GPU资源进行推断时提供最佳的推断延迟。而使用最大GPU资源量调优的模型在GPU资源受限时推理延迟较差。另一方面，使用适当数量的GPU资源进行调整的模型仍然可以在广泛的GPU资源可用性范围内实现良好的推理延迟。我们探讨了在不同GPU资源量下影响模型调整的原因。我们提出了许多技术，以最大限度地提高资源利用率，提高调优性能。我们启用GPU的受控空间共享，以在GPU上复用多个调优应用程序。我们扩展了调优服务器实例，并将调优模型分割到多个客户端实例中，以便同时调优模型的不同运算符，从而实现更好的GPU复用。通过我们的改进，我们将DNN自动调整时间减少了75%，并将吞吐量提高了5倍。
英文摘要：
GPUs are used for training, inference, and tuning the machine learning models. However, Deep Neural Network (DNN) vary widely in their ability to exploit the full power of high-performance GPUs. Spatial sharing of GPU enables multiplexing several DNNs on the GPU and can improve GPU utilization, thus improving throughput and lowering latency. DNN models given just the right amount of GPU resources can still provide low inference latency, just as much as dedicating all of the GPU for their inference task. An approach to improve DNN inference is tuning of the DNN model. Autotuning frameworks find the optimal low-level implementation for a certain target device based on the trained machine learning model, thus reducing the DNN's inference latency and increasing inference throughput. We observe an interdependency between the tuned model and its inference latency. A DNN model tuned with specific GPU resources provides the best inference latency when inferred with close to the same amount of GPU resources. While a model tuned with the maximum amount of the GPU's resources has poorer inference latency once the GPU resources are limited for inference. On the other hand, a model tuned with an appropriate amount of GPU resources still achieves good inference latency across a wide range of GPU resource availability. We explore the causes that impact the tuning of a model at different amounts of GPU resources. We present many techniques to maximize resource utilization and improve tuning performance. We enable controlled spatial sharing of GPU to multiplex several tuning applications on the GPU. We scale the tuning server instances and shard the tuning model across multiple client instances for concurrent tuning of different operators of a model, achieving better GPU multiplexing. With our improvements, we decrease DNN autotuning time by up to 75 percent and increase throughput by a factor of 5.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
