arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2008.09491cs.DC
收藏

面向设计自主管理的机器学习推理服务系统inPublic Cloud

Towards Designing a Self-Managed Machine Learning Inference Serving System inPublic Cloud

Jashwant Raj Gunasekaran, Prashanth Thinakaran, Cyan Subhra Mishra, Mahmut Taylan Kandemir, Chita R. Das

摘要:

我们正在见证使用基于机器学习(ML)的预测系统的增长趋势,这些系统跨越不同的应用领域,包括产品推荐系统、个人助理设备、面部识别等。这些应用通常在准确性和响应延迟方面有不同的要求,这直接影响了在公共云中部署它们的成本。此外,部署成本还取决于所采购的资源类型,这些资源本身在配置延迟和计费复杂性方面是异构的。因此,对于推理服务系统来说,从这种混杂的资源类型和模型类型阵列中进行选择以提供低延迟和成本有效的推理是很困难的。在这项工作中,我们定量地描述了在不同的公共云资源产品上托管ML推理的成本,准确性和延迟影响。此外,我们谨慎地评估先前的工作,试图提高成本效益的预测服务。我们的评估表明,现有的工作并没有从模型和资源异构性两个维度解决问题。因此,我们认为,要解决这个问题,我们需要从整体上解决的问题时,试图结合模型和资源异构性优化应用程序的约束,为此,我们设想开发一个自我管理的推理服务系统,它可以优化应用程序的需求,基于公共云资源的特点。为了解决这个复杂的优化问题,我们探索了一个基于学习的系统的高层设计,该系统可以有效地适应大规模系统不断变化的需求。

英文摘要:

We are witnessing an increasing trend towardsusing Machine Learning (ML) based prediction systems, span-ning across different application domains, including productrecommendation systems, personal assistant devices, facialrecognition, etc. These applications typically have diverserequirements in terms of accuracy and response latency, thathave a direct impact on the cost of deploying them in a publiccloud. Furthermore, the deployment cost also depends on thetype of resources being procured, which by themselves areheterogeneous in terms of provisioning latencies and billingcomplexity. Thus, it is strenuous for an inference servingsystem to choose from this confounding array of resourcetypes and model types to provide low-latency and cost-effectiveinferences. In this work we quantitatively characterize the cost,accuracy and latency implications of hosting ML inferenceson different public cloud resource offerings. In addition, wecomprehensively evaluate prior work which tries to achievecost-effective prediction-serving. Our evaluation shows that,prior work does not solve the problem from both dimensionsof model and resource heterogeneity. Hence, we argue that toaddress this problem, we need to holistically solve the issuesthat arise when trying to combine both model and resourceheterogeneity towards optimizing for application constraints.Towards this, we envision developing a self-managed inferenceserving system, which can optimize the application require-ments based on public cloud resource characteristics. In orderto solve this complex optimization problem, we explore the highlevel design of a reinforcement-learning based system that canefficiently adapt to the changing needs of the system at scale.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑