arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.14066cs.AIcs.PF
收藏

TurboSpec:用于优化 LLM 服务有效吞吐的闭环推测控制系统

TurboSpec: Closed-loop Speculation Control System for Optimizing LLM Serving Goodput

UC Berkeley(伯克利大学);UCSD(圣地亚哥大学);Tsinghua University(清华大学);University of Chicago(芝加哥大学);SJTU(上海交通大学)

Xiaoxuan Liu, Jongseok Park, Langxiang Hu, Woosuk Kwon, Zhuohan Li, Chen Zhang, Kuntai Du, Xiangxi Mo, Kaichao You, Alvin Cheung, Zhijie Deng, Ion Stoica, Hao Zhang

AI总结:

针对推测解码在 LLM 服务中收益不稳定的问题,提出 TurboSpec 闭环控制系统,自动剖析环境并基于反馈动态调节请求内并行度,在 vLLM 多场景中实现一致性能提升。

摘要:

大语言模型(LLM)服务系统通过批处理并发用户请求来实现高效服务。然而,在实际部署中,这种来自批处理的请求间并行性往往受到低请求率或内存限制等外部因素的制约。近期工作将推测解码所带来的请求内并行性作为解决该问题的方案。遗憾的是,请求内并行性带来的收益通常很脆弱,因为推测解码会产生开销,而且推测出的令牌可能未命中。我们观察到,如果不根据传入请求和推测方法进行调优,只是简单地加入推测解码,反而可能降低 LLM 服务性能。为了减少对专家调优的需求并增强推测解码的稳健性,我们提出了 TurboSpec,这是一个推测控制系统,可自动分析执行环境,并利用基于反馈的算法动态调整 LLM 服务中请求内并行性的程度。TurboSpec 会预测“goodput(有效吞吐)”——即成功生成令牌的数量——以评估请求内并行性,并在运行时将其调整到具有最高 goodput 的水平。我们在实际 LLM 服务系统 vLLM 上实现了 TurboSpec,并在多种工作负载和硬件配置下证明了其有效性,在所有测试场景中均带来了一致的性能提升。

英文摘要:

Large Language Model (LLM) serving systems batch concurrent user requests to achieve efficient serving. However, in real-world deployments, such inter-request parallelism from batching is often limited by external factors such as low request rates or memory constraints. Recent works focus on intra-request parallelism from speculative decoding as a solution to this problem. Unfortunately, benefits from intra-request parallelism are often fragile, as speculative decoding causes overhead, and speculated tokens may miss. We observe that speculative decoding may degrade LLM serving performance if added naively without tuning to the incoming requests and the speculation method. To alleviate the need for expert tuning and make speculative decoding more robust, we present TurboSpec, a speculation control system that automatically profiles the execution environment and utilizes a feedback-based algorithm to dynamically adjust the amount of intra-request parallelism in LLM serving. TurboSpec predicts "goodput" - the amount of successfully generated tokens - to evaluate and adjust intra-request parallelism amount to that with the highest goodput in runtime. We implement TurboSpec on a real-world LLM serving system vLLM and demonstrate its effectiveness across diverse workloads and hardware configurations, providing consistent performance improvements across all test scenarios.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑