arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2304.10892cs.LGcs.DCcs.SYeess.SY
收藏

协调推理服务系统的高准确性、成本效率与低延迟

Reconciling High Accuracy, Cost-Efficiency, and Low Latency of Inference Serving Systems

Iran University of Science and Technology(伊朗科技大学);Queen Mary University of London(伦敦玛丽女王大学);Technical University of Darmstadt(达姆施塔特工业大学);University of South Carolina(南卡罗来纳大学)

Mehran Salmani, Saeid Ghafouri, Alireza Sanaee, Kamran Razavi, Max Mühlhäuser, Joseph Doyle, Pooyan Jamshidi, Mohsen Sharifi

AI总结:

针对动态工作负载下ML推理服务难以兼顾准确性、延迟与成本的问题,提出InfAdapter,通过主动选择模型变体及资源分配以最大化准确性与成本目标,显著降低SLO违规与成本。

摘要:

机器学习(ML)推理在各类应用中的使用正急剧增长。ML推理服务直接与用户交互,需要快速且准确的响应。此外,这些服务面临动态的请求工作负载,迫使其计算资源发生改变。未能合理调整计算资源的规模会导致延迟服务级别目标(SLOs)违规或计算资源浪费。在兼顾准确性、延迟和资源成本所有支柱的情况下适应动态工作负载极具挑战。为应对这些挑战,我们提出了InfAdapter,它主动选择一组ML模型变体及其资源分配,以满足延迟SLO,同时最大化由准确性和成本组成的目标函数。与流行的行业自动扩缩容工具相比,InfAdapter分别将SLO违规和成本最多降低了65%和33%。

英文摘要:

The use of machine learning (ML) inference for various applications is growing drastically. ML inference services engage with users directly, requiring fast and accurate responses. Moreover, these services face dynamic workloads of requests, imposing changes in their computing resources. Failing to right-size computing resources results in either latency service level objectives (SLOs) violations or wasted computing resources. Adapting to dynamic workloads considering all the pillars of accuracy, latency, and resource cost is challenging. In response to these challenges, we propose InfAdapter, which proactively selects a set of ML model variants with their resource allocations to meet latency SLO while maximizing an objective function composed of accuracy and cost. InfAdapter decreases SLO violation and costs up to 65% and 33%, respectively, compared to a popular industry autoscaler (Kubernetes Vertical Pod Autoscaler).

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑