PRETZEL:打开机器学习预测服务系统的黑箱
PRETZEL: Opening the Black Box of Machine Learning Prediction Serving Systems
摘要:
机器学习模型通常由转换管道组成。虽然这种设计允许在训练时有效地执行单个模型组件,但预测服务具有不同的要求,例如低延迟,高吞吐量和重负载下的适度性能下降。当前的预测服务系统将模型视为黑盒,从而忽略了预测时间特定的优化以便于部署。在本文中,我们提出了PRETZEL,预测服务系统引入了一种新的白盒架构,使端到端和多模型优化。使用类似生产的模型管道,我们的实验表明,PRETZEL能够在不同的维度上引入性能改进;与最先进的方法相比,PRETZEL平均能够将第99百分位延迟减少5.5倍,同时将内存占用减少25倍,并将吞吐量增加4.7倍。
英文摘要:
Machine Learning models are often composed of pipelines of transformations. While this design allows to efficiently execute single model components at training time, prediction serving has different requirements such as low latency, high throughput and graceful performance degradation under heavy load. Current prediction serving systems consider models as black boxes, whereby prediction-time-specific optimizations are ignored in favor of ease of deployment. In this paper, we present PRETZEL, a prediction serving system introducing a novel white box architecture enabling both end-to-end and multi-model optimizations. Using production-like model pipelines, our experiments show that PRETZEL is able to introduce performance improvements over different dimensions; compared to state-of-the-art approaches PRETZEL is on average able to reduce 99th percentile latency by 5.5x while reducing memory footprint by 25x, and increasing throughput by 4.7x.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。