arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1905.13348cs.DCcs.LG
收藏

INFaaS:一个无模型可管理推理服务系统

INFaaS: A Model-less and Managed Inference Serving System

Stanford University(斯坦福大学)

Francisco Romero, Qian Li, Neeraja J. Yadwadkar, Christos Kozyrakis

摘要:

尽管在机器学习推理服务方面已有工作,但易用性和成本效率在大规模上仍然面临挑战。开发人员必须手动搜索数千个模型变体-已经训练的模型的版本,这些版本在硬件,资源占用,延迟,成本和准确性方面有所不同-以满足不同的应用程序需求。由于需求、查询负载和应用程序本身会随着时间的推移而发展,因此需要为每个推理查询动态地做出这些决策,以避免通过简单的自动缩放产生过多的成本。为了避免在模型变量的大而复杂的权衡空间中导航,开发人员经常在查询中修复一个变量,并在负载增加时复制它。然而,考虑到云中变体和硬件平台的多样性,缺乏对权衡空间的理解可能会给开发人员带来巨大的成本。本文介绍了INFaaS,分布式推理服务的管理和无模型系统,开发人员只需指定其应用程序的性能和准确性要求,而不需要为每个查询指定一个特定的模型变量。INFaaS生成模型变量,并代表开发人员有效地导航模型变量的大权衡空间,以满足特定于应用程序的目标:(a)对于每个查询,它选择一个模型,硬件架构和模型优化,(b)它将VM级水平自动缩放与模型级自动缩放相结合,其中多个不同的模型变量用于在每台机器内服务查询。与AWS EC2上最先进的推理服务系统相比,通过利用不同的变体并跨模型共享硬件资源,INFaaS实现了1.3倍的吞吐量,违反延迟目标的频率降低了1.6倍,并节省了高达21.6倍的成本(平均为8.5倍)。

英文摘要:

Despite existing work in machine learning inference serving, ease-of-use and cost efficiency remain challenges at large scales. Developers must manually search through thousands of model-variants -- versions of already-trained models that differ in hardware, resource footprints, latencies, costs, and accuracies -- to meet the diverse application requirements. Since requirements, query load, and applications themselves evolve over time, these decisions need to be made dynamically for each inference query to avoid excessive costs through naive autoscaling. To avoid navigating through the large and complex trade-off space of model-variants, developers often fix a variant across queries, and replicate it when load increases. However, given the diversity across variants and hardware platforms in the cloud, a lack of understanding of the trade-off space can incur significant costs to developers. This paper introduces INFaaS, a managed and model-less system for distributed inference serving, where developers simply specify the performance and accuracy requirements for their applications without needing to specify a specific model-variant for each query. INFaaS generates model-variants, and efficiently navigates the large trade-off space of model-variants on behalf of developers to meet application-specific objectives: (a) for each query, it selects a model, hardware architecture, and model optimizations, (b) it combines VM-level horizontal autoscaling with model-level autoscaling, where multiple, different model-variants are used to serve queries within each machine. By leveraging diverse variants and sharing hardware resources across models, INFaaS achieves 1.3x higher throughput, violates latency objectives 1.6x less often, and saves up to 21.6x in cost (8.5x on average) compared to state-of-the-art inference serving systems on AWS EC2.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑