WISP: 通过动态草稿和SLO感知批处理实现边缘的废料和干扰抑制的分布式推测LLM服务
WISP: Waste- and Interference-Suppressed Distributed Speculative LLM Serving at the Edge via Dynamic Drafting and SLO-Aware Batching
AI总结:
本文提出WISP系统,通过动态草稿和SLO感知批处理优化边缘和云的负载平衡,提升系统容量和吞吐量。
摘要:
随着大型语言模型(LLM)越来越被终端用户访问,越来越多的推理请求由边缘设备发起并在集中式GPU集群上计算。然而,由此导致的计算负载指数增长对数据中心造成了巨大压力,而边缘设备则大多未被充分利用,导致网络内负载和资源效率不平衡。通过推测解码将边缘设备整合到LLM推理过程中有助于平衡边缘和云之间的负载,同时保持无损的预测准确性。本文识别并正式化了两个限制分布式推测LLM服务效率和可扩展性的关键瓶颈:浪费的草稿时间和验证干扰。为了解决这些挑战,我们提出了WISP,一个高效且SLO感知的分布式LLM推理系统,包括智能推测控制器、验证时间估计器和验证批调度器。这些组件协同提高草稿效率并优化服务器上的验证请求调度。广泛的结果表明,与集中式服务和SLED相比,WISP将系统容量提高了最多2.1倍和4.1倍,并将系统吞吐量提高了最多1.94倍和3.7倍。
英文摘要:
As Large Language Models (LLMs) become increasingly accessible to end users, an ever-growing number of inference requests are initiated from edge devices and computed on centralized GPU clusters. However, the resulting exponential growth in computation workload is placing significant strain on data centers, while edge devices remain largely underutilized, leading to imbalanced workloads and resource inefficiency across the network. Integrating edge devices into the LLM inference process via speculative decoding helps balance the workload between the edge and the cloud, while maintaining lossless prediction accuracy. In this paper, we identify and formalize two critical bottlenecks that limit the efficiency and scalability of distributed speculative LLM serving: Wasted Drafting Time and Verification Interference. To address these challenges, we propose WISP, an efficient and SLO-aware distributed LLM inference system that consists of an intelligent speculation controller, a verification time estimator, and a verification batch scheduler. These components collaboratively enhance drafting efficiency and optimize verification request scheduling on the server. Extensive numerical results show that WISP improves system capacity by up to 2.1x and 4.1x, and increases system goodput by up to 1.94x and 3.7x, compared to centralized serving and SLED, respectively.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。