SCOOT:面向 LLM 推理引擎的 SLO 性能调优
SCOOT: SLO-Oriented Performance Tuning for LLM Inference Engines
AI总结:
本文提出自动性能调优系统 SCOOT,通过单/多目标贝叶斯优化、约束剪枝、随机森林学习隐藏约束及并行建议,为不同 LLM 推理引擎优化 SLO,并已在蚂蚁集团生产环境落地。
摘要:
随着大语言模型(LLM)在各类 Web 应用中日益普及,优化 LLM 推理服务的服务等级目标(SLO)对于提升用户满意度和增强云厂商竞争力具有重要意义。本文观察到,调整 LLM 推理引擎的参数能够改善服务性能,而且不同服务的最优参数配置并不相同。因此,我们提出 SCOOT,这是一个自动性能调优系统,通过调节推理引擎参数,为每个 LLM 推理服务优化 SLO。SCOOT 联合利用单目标和多目标贝叶斯优化(BO)技术,通过探索与利用处理各种优化目标。此外,SCOOT 使用已知约束裁剪搜索空间,并采用随机森林在调优过程中学习隐藏约束,以减少无效探索。为提高调优效率,SCOOT 利用并行建议加速调优过程。大量实验表明,SCOOT 在 SLO 优化方面显著优于现有调优技术,同时大幅提高了调优效率。此外,SCOOT 可普遍适用于包括 vLLM 和 TensorRT-LLM 在内的多种 LLM 推理引擎。目前,SCOOT 已在蚂蚁集团的生产环境中部署实现。
英文摘要:
As large language models (LLMs) are gaining increasing popularity across a wide range of web applications, it is of great importance to optimize service-level objectives (SLOs) for LLM inference services to enhance user satisfaction and improve the competitiveness of cloud vendors. In this paper, we observe that adjusting the parameters of LLM inference engines can improve service performance, and the optimal parameter configurations of different services are different. Therefore, we propose SCOOT, an automatic performance tuning system to optimize SLOs for each LLM inference service by tuning the parameters of the inference engine. SCOOT jointly exploits single-objective and multiple-objective Bayesian optimization (BO) techniques to handle various optimization objectives via exploration and exploitation. Moreover, SCOOT prunes the search space with known constraints and adopts a random forest to learn hidden constraints during the tuning process to mitigate invalid exploration. To improve the tuning efficiency, SCOOT utilizes the parallel suggestion to accelerate the tuning process. Extensive experiments demonstrate that SCOOT considerably outperforms existing tuning techniques in SLO optimization while greatly improving the tuning efficiency. Moreover, SCOOT is universally applicable to various LLM inference engines including vLLM and TensorRT-LLM. Currently, SCOOT has already been implemented in the production environment at Ant Group.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。