Orloj:可预测地服务不可预测的DNN
Orloj: Predictably Serving Unpredictable DNNs
摘要:
现有的DNN服务解决方案可以提供紧凑的延迟SLO,同时通过仔细调度传入请求来保持高吞吐量,其执行时间被假定为高度可预测且与数据无关。然而,对新兴动态DNN的推理请求-例如,流行的自然语言处理(NLP)模型和跳过层的计算机视觉(CV)模型是数据依赖的。当使用现有的解决方案时,它们表现出很差的性能,因为它们在请求执行时间上会有很大的差异,这取决于输入--批处理中最长的请求会增加较小请求的执行时间,在没有仔细检查的情况下导致SLO失败。在本文中,我们提出了Orloj,一个动态DNN服务系统,它使用预期请求执行时间的经验分布来捕获动态DNN中的这种变化,然后在不知道请求的精确执行时间的情况下有效地批处理和调度它们。Orloj在严格的SLO约束下的完成率明显优于最先进的高方差动态DNN工作负载服务解决方案51- 80%,在更宽松的SLO设置下超过100%。对于经过充分研究的静态DNN工作负载,Orloj保持了与最先进技术相当的性能。
英文摘要:
Existing DNN serving solutions can provide tight latency SLOs while maintaining high throughput via careful scheduling of incoming requests, whose execution times are assumed to be highly predictable and data-independent. However, inference requests to emerging dynamic DNNs -- e.g., popular natural language processing (NLP) models and computer vision (CV) models that skip layers -- are data-dependent. They exhibit poor performance when served using existing solutions because they experience large variance in request execution times depending on the input -- the longest request in a batch inflates the execution times of the smaller ones, causing SLO misses in the absence of careful batching. In this paper, we present Orloj, a dynamic DNN serving system, that captures this variance in dynamic DNNs using empirical distributions of expected request execution times, and then efficiently batches and schedules them without knowing a request's precise execution time. Orloj significantly outperforms state-of-the-art serving solutions for high variance dynamic DNN workloads by 51--80% in finish rate under tight SLO constraints, and over 100% under more relaxed SLO settings. For well-studied static DNN workloads, Orloj keeps comparable performance with the state-of-the-art.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。