arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.14843cs.DC
收藏

两种尺度的故事:调和推理服务系统中的水平与垂直扩展

A Tale of Two Scales: Reconciling Horizontal and Vertical Scaling for Inference Serving Systems

Kamran Razavi, Mehran Salmani, Max Mühlhäuser, Boris Koldehofe, Lin Wang

AI总结:

本文提出Themis系统,结合水平与垂直扩展的两阶段自动扩展策略,通过动态规划优化推理服务资源,实现SLO违规降低超10倍并保持资源效率。

摘要:

推理服务在将机器学习模型部署到实际应用中至关重要,确保对推理请求的高效处理和快速响应。然而,管理这些系统中的资源带来了重大挑战,特别是在多变且不可预测的工作负载下保持性能。两种主要的扩展策略,即水平扩展和垂直扩展,提供了不同的优势和局限性。水平扩展通过增加更多实例来处理增加的负载,但可能遭受冷启动问题和管理复杂性的增加。垂直扩展提升现有实例的容量,允许更快的响应,但受限于硬件和模型并行化能力。本文介绍了Themis,一个旨在利用推理服务系统中水平与垂直扩展两者优势的系统。Themis采用两阶段自动扩展策略:首先使用原地垂直扩展来处理工作负载激增,然后在工作负载稳定后切换到水平扩展以优化资源效率。该系统对深度学习模型的处理延迟进行画像,计算排队延迟,并采用不同的动态规划算法,根据工作负载情况最优地解决联合水平与垂直扩展问题。使用真实工作负载轨迹的广泛评估表明,与最先进的水平或垂直自动扩展方法相比,SLO违规减少了超过10倍,同时在工作负载稳定时保持资源效率。

英文摘要:

Inference serving is of great importance in deploying machine learning models in real-world applications, ensuring efficient processing and quick responses to inference requests. However, managing resources in these systems poses significant challenges, particularly in maintaining performance under varying and unpredictable workloads. Two primary scaling strategies, horizontal and vertical scaling, offer different advantages and limitations. Horizontal scaling adds more instances to handle increased loads but can suffer from cold start issues and increased management complexity. Vertical scaling boosts the capacity of existing instances, allowing for quicker responses but is limited by hardware and model parallelization capabilities. This paper introduces Themis, a system designed to leverage the benefits of both horizontal and vertical scaling in inference serving systems. Themis employs a two-stage autoscaling strategy: initially using in-place vertical scaling to handle workload surges and then switching to horizontal scaling to optimize resource efficiency once the workload stabilizes. The system profiles the processing latency of deep learning models, calculates queuing delays, and employs different dynamic programming algorithms to solve the joint horizontal and vertical scaling problem optimally based on the workload situation. Extensive evaluations with real-world workload traces demonstrate over $10\times$ SLO violation reduction compared to the state-of-the-art horizontal or vertical autoscaling approaches while maintaining resource efficiency when the workload is stable.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑