像发条一样服务DNN：自底向上的性能可预测性
Serving DNNs like Clockwork: Performance Predictability from the Bottom Up
摘要：
机器学习推理正在成为交互式Web应用程序的核心构建块。因此，这些应用程序所依赖的底层模型服务系统必须始终满足低延迟目标。现有的模型服务架构使用众所周知的反应式技术来缓解常见的延迟源，但不能有效地减少不可预测的执行时间引起的尾部延迟。然而，底层的执行时间并不是根本不可预测的-相反，我们观察到使用深度神经网络（DNN）模型的推理具有确定性性能。在这里，从各个DNN推理的可预测执行时间开始，我们采用了一种原则性的设计方法，成功地构建了一个完全分布式的模型服务系统，实现了可预测的端到端性能。我们使用生产跟踪工作负载评估了我们的实现Clockwork，并表明Clockwork可以支持数千种模型，同时满足99.9999%请求的100 ms延迟目标。我们进一步证明了Clockwork利用可预测的执行时间来实现严格的请求级服务级目标（SLO）以及高度的请求级性能隔离。
英文摘要：
Machine learning inference is becoming a core building block for interactive web applications. As a result, the underlying model serving systems on which these applications depend must consistently meet low latency targets. Existing model serving architectures use well-known reactive techniques to alleviate common-case sources of latency, but cannot effectively curtail tail latency caused by unpredictable execution times. Yet the underlying execution times are not fundamentally unpredictable - on the contrary we observe that inference using Deep Neural Network (DNN) models has deterministic performance. Here, starting with the predictable execution times of individual DNN inferences, we adopt a principled design methodology to successively build a fully distributed model serving system that achieves predictable end-to-end performance. We evaluate our implementation, Clockwork, using production trace workloads, and show that Clockwork can support thousands of models while simultaneously meeting 100ms latency targets for 99.9999% of requests. We further demonstrate that Clockwork exploits predictable execution times to achieve tight request-level service-level objectives (SLOs) as well as a high degree of request-level performance isolation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
