arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2401.11240cs.DC
收藏

CaraServe:面向生成式LLM推理的CPU辅助且秩感知的LoRA服务系统

CaraServe: CPU-Assisted and Rank-Aware LoRA Serving for Generative LLM Inference

Suyi Li, Hanfeng Lu, Tianyuan Wu, Minchen Yu, Qizhen Weng, Xusheng Chen, Yizhou Shan, Binhang Yuan, Wei Wang

AI总结:

CaraServe通过CPU辅助预填充和秩感知调度,解决多LoRA适配器服务中GPU冷启动延迟问题,实现最高1.4倍加速和99%的SLO达成率。

摘要:

预训练大语言模型(LLMs)通常需要针对特定领域任务进行专门化。低秩适配(LoRA)是一种流行的方法,通过添加轻量级可训练适配器使基础模型适配多个任务。在本文中,我们提出了CaraServe,一个高效服务源自共同基础模型的众多LoRA适配器的系统。CaraServe将基础模型保持在GPU上,并从主内存动态加载被激活的LoRA适配器。由于GPU加载会导致冷启动,显著延迟token生成,CaraServe采用了一种CPU辅助的方法。它在被激活的适配器加载到GPU上的同时,提前在CPU上启动这些适配器进行预填充;加载完成后,再切换到GPU上进行生成式LoRA推理。CaraServe开发了一种高度优化的同步机制,以高效协调CPU和GPU上的LoRA计算。此外,CaraServe采用了一种秩感知的调度算法,以最优方式调度异构LoRA请求,从而最大化服务水平目标(SLO)达成率。我们已实现CaraServe,并针对最先进的LoRA服务系统进行了评估。实验结果表明,CaraServe可以将平均请求服务延迟加速高达1.4倍,并实现高达99%的SLO达成率。

英文摘要:

Pre-trained large language models (LLMs) often need specialization for domain-specific tasks. Low-Rank Adaptation (LoRA) is a popular approach that adapts a base model to multiple tasks by adding lightweight trainable adapters. In this paper, we present CaraServe, a system that efficiently serves many LoRA adapters derived from a common base model. CaraServe maintains the base model on GPUs and dynamically loads activated LoRA adapters from main memory. As GPU loading results in a cold-start that substantially delays token generation, CaraServe employs a CPU-assisted approach. It early starts the activated adapters on CPUs for prefilling as they are being loaded onto GPUs; after loading completes, it then switches to the GPUs for generative LoRA inference. CaraServe develops a highly optimized synchronization mechanism to efficiently coordinate LoRA computation on the CPU and GPU. Moreover, CaraServe employs a rank-aware scheduling algorithm to optimally schedule heterogeneous LoRA requests for maximum service-level objective (SLO) attainment. We have implemented CaraServe and evaluated it against state-of-the-art LoRA serving systems. Our results demonstrate that CaraServe can speed up the average request serving latency by up to 1.4$\times$ and achieve an SLO attainment of up to 99%.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑