iServe:一种面向LLM的基于意图的服务系统
iServe: An Intent-based Serving System for LLMs
AI总结:
iServe是一个基于意图的分布式LLM推理系统,通过轻量级指纹估计配置影响,动态选择最优配置,显著降低延迟、SLO违规和剖析成本,同时提升GPU吞吐量。
摘要:
大型语言模型(LLMs)在各行业中正变得无处不在,应用程序要求它们满足多样化的用户意图。然而,开发者目前面临手动探索众多部署配置的挑战——这些配置是影响资源使用、延迟、成本和准确性的并行与压缩技术的组合——以满足这些意图。评估这些配置对用户指标的影响需要对每个模型进行广泛且昂贵的性能剖析。现有方法通过使用固定、静态的配置来避免这种开销,但这往往导致次优的性能和更高的成本。此外,这些解决方案均无法动态适应变化的用户意图,以有效平衡延迟和成本。我们提出了iServe,一个自动化的、基于意图的分布式LLM推理系统。开发者无需手动选择部署配置,只需指定其意图——例如最小化延迟、降低成本或满足任一方面的特定目标。iServe引入了指纹(fingerprints),即LLM的轻量级表示,以高效估计不同配置对延迟和内存使用的影响。基于这些洞察和GPU可用性,iServe动态选择最优配置以适应用户的意图。对于各种LLM和查询到达率,与最先进的系统相比,iServe最能满足用户意图,将延迟降低77.62%,SLO违规减少7.09倍,同时将GPU吞吐量提高4.72倍。此外,与基线相比,iServe基于指纹的性能剖析将剖析成本(GPU小时)降低了6.05倍。
英文摘要:
Large Language Models (LLMs) are becoming ubiquitous across industries, where applications demand they fulfill diverse user intents. However, developers currently face the challenge of manually exploring numerous deployment configurations - combinations of parallelism and compression techniques that impact resource usage, latency, cost, and accuracy - to meet these intents. Assessing the impact of these configurations on user metrics requires extensive, costly profiling for each model. Existing approaches avoid this expense by using fixed, static configurations, but this often leads to sub-optimal performance and higher costs. Moreover, none of these solutions dynamically adapt to changing user intents to balance latency and cost, effectively. We present iServe, an automated, intent-based system for distributed LLM inference. Instead of manually selecting deployment configurations, developers simply specify their intent - such as minimizing latency, reducing cost, or meeting specific targets for either. iServe introduces fingerprints, lightweight representations of LLMs, to efficiently estimate how different configurations impact latency and memory usage. Based on these insights and GPU availability, iServe dynamically selects the optimal configuration to align with the user's intent. For various LLMs and query arrival rates, iServe best meets user intents compared to state-of-the-art systems by reducing latency by 77.62% and SLO violations by 7.09x while improving GPU throughput by 4.72x. Moreover, iServe's fingerprint-based profiling reduces profiling cost by 6.05x (GPU-hours) compared to baselines.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。