arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2106.05345cs.DCcs.LG
收藏

鸡尾酒:利用Ensemble Learning优化公共云中的模型服务

Cocktail: Leveraging Ensemble Learning for Optimized Model Serving in Public Cloud

Pennsylvania State University(宾夕法尼亚州立大学)

Jashwant Raj Gunasekaran, Cyan Subhra Mishra, Prashanth Thinakaran, Mahmut Taylan Kandemir, Chita R. Das

摘要:

随着对各种应用服务采用ML模型的需求不断增长,服务于这些模型的框架能够以最小的延迟提供高度准确的预测以及降低公共云环境中的部署成本至关重要。尽管高延迟,在这一领域的先前工作是至关重要的限制由个别模型提供的准确性。直观地说,模型集成可以通过智能地并行组合不同的模型来解决精度差距。然而,在运行时动态地选择合适的模型以满足期望的准确性和低延迟,以最小的部署成本是一个不平凡的问题。Cocktail是一个基于集成的模型服务框架,它包括两个关键部分:(1)动态模型选择框架,它减少了集成中的模型数量,同时满足精度和延迟要求;(ii)采用与重要性采样组合的分布式主动自动缩放策略的自适应资源管理(RM)框架,为模特们有效地分配资源。RM框架存储临时虚拟机(VM)实例以降低公共云中的部署成本。Cocktail在AWS EC2平台上的原型实现和使用各种工作负载的详尽评估表明,与最先进的模型服务框架相比,Cocktail可以将部署成本降低1.45倍,同时将延迟降低2倍,并满足高达96%的请求的目标准确性。

英文摘要:

With a growing demand for adopting ML models for a varietyof application services, it is vital that the frameworks servingthese models are capable of delivering highly accurate predic-tions with minimal latency along with reduced deploymentcosts in a public cloud environment. Despite high latency,prior works in this domain are crucially limited by the accu-racy offered by individual models. Intuitively, model ensem-bling can address the accuracy gap by intelligently combiningdifferent models in parallel. However, selecting the appro-priate models dynamically at runtime to meet the desiredaccuracy with low latency at minimal deployment cost is anontrivial problem. Towards this, we proposeCocktail, a costeffective ensembling-based model serving framework.Cock-tailcomprises of two key components: (i) a dynamic modelselection framework, which reduces the number of modelsin the ensemble, while satisfying the accuracy and latencyrequirements; (ii) an adaptive resource management (RM)framework that employs a distributed proactive autoscalingpolicy combined with importance sampling, to efficiently allo-cate resources for the models. The RM framework leveragestransient virtual machine (VM) instances to reduce the de-ployment cost in a public cloud. A prototype implementationofCocktailon the AWS EC2 platform and exhaustive evalua-tions using a variety of workloads demonstrate thatCocktailcan reduce deployment cost by 1.45x, while providing 2xreduction in latency and satisfying the target accuracy for upto 96% of the requests, when compared to state-of-the-artmodel-serving frameworks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑