arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2210.05889cs.DCcs.LG
收藏

KAIROS:利用异构云资源构建具有成本效益的机器学习推理系统

KAIROS: Building Cost-Efficient Machine Learning Inference Systems with Heterogeneous Cloud Resources

Northeastern University(东北大学);MIT(麻省理工学院)

Baolin Li, Siddharth Samsi, Vijay Gadepally, Devesh Tiwari

AI总结:

针对云平台上在线推理服务的QoS与成本约束问题,提出新型运行时框架KAIROS,通过无开销构建异构硬件池、最优分配查询,实现吞吐量较同构方案最高提升1倍、较现有最优方案最高提升70%的效果。

摘要:

在线推理正成为众多企业的核心服务产品,部署在云平台以满足客户需求。尽管这类服务具备创收能力,但仍需在严格的服务质量(QoS)与成本预算约束下运行。本文提出KAIROS,这是一种新型运行时框架,可在满足QoS目标与成本预算的前提下最大化查询吞吐量。KAIROS设计并实现了多项新颖技术,可在无在线探索开销的情况下构建异构计算硬件池,并在运行时最优分配推理查询。我们采用工业级深度学习(DL)模型开展评估,结果显示KAIROS的吞吐量最高可达最优同构方案的2倍,且性能优于现有最优方案最高达70%——即便我们为对比方案提供了有利设置,忽略了它们的探索开销。

英文摘要:

Online inference is becoming a key service product for many businesses, deployed in cloud platforms to meet customer demands. Despite their revenue-generation capability, these services need to operate under tight Quality-of-Service (QoS) and cost budget constraints. This paper introduces KAIROS, a novel runtime framework that maximizes the query throughput while meeting QoS target and a cost budget. KAIROS designs and implements novel techniques to build a pool of heterogeneous compute hardware without online exploration overhead, and distribute inference queries optimally at runtime. Our evaluation using industry-grade deep learning (DL) models shows that KAIROS yields up to 2X the throughput of an optimal homogeneous solution, and outperforms state-of-the-art schemes by up to 70%, despite advantageous implementations of the competing schemes to ignore their exploration overhead.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑