arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1912.06322cs.PFcs.LG
收藏

基于GPU的动态批处理推理服务器的性能分析:一个封闭形式的特征

Queueing Analysis of GPU-Based Inference Servers with Dynamic Batching: A Closed-Form Characterization

Osaka University(大阪大学)

Yoshiaki Inoue

摘要:

GPU加速计算是利用深度神经网络(DNN)实现高速推理服务器的关键技术。基于GPU的推理的一个重要特征是,在处理速度和能耗方面,通过批量处理多个作业,计算效率大幅提高。在本文中,我们制定了基于GPU的推理服务器作为一个批处理模型的批量大小依赖的处理时间。我们首先表明,服务器的能源效率随着推理作业的到达率单调增加,这表明在推理作业的延迟要求内尽可能高的利用率水平下操作推理服务器是节能的。然后,我们推导出一个封闭形式的平均延迟,这提供了一个简单的表征的延迟性能的上限。通过模拟和数值实验,我们表明,平均延迟的精确值很好地近似于这个上限。我们进一步比较了这个上限与延迟曲线的实际实现的基于GPU的推理服务器,我们表明,真正的性能曲线很好地解释了派生的简单公式。

英文摘要:

GPU-accelerated computing is a key technology to realize high-speed inference servers using deep neural networks (DNNs). An important characteristic of GPU-based inference is that the computational efficiency, in terms of the processing speed and energy consumption, drastically increases by processing multiple jobs together in a batch. In this paper, we formulate GPU-based inference servers as a batch service queueing model with batch-size dependent processing times. We first show that the energy efficiency of the server monotonically increases with the arrival rate of inference jobs, which suggests that it is energy-efficient to operate the inference server under a utilization level as high as possible within a latency requirement of inference jobs. We then derive a closed-form upper bound for the mean latency, which provides a simple characterization of the latency performance. Through simulation and numerical experiments, we show that the exact value of the mean latency is well approximated by this upper bound. We further compare this upper bound with the latency curve measured in real implementation of GPU-based inference servers and we show that the real performance curve is well explained by the derived simple formula.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑