PICE:云边网络中面向LLM服务的语义驱动渐进式推理系统
PICE: A Semantic-Driven Progressive Inference System for LLM Serving in Cloud-Edge Networks
AI总结:
针对大语言模型推理成本高、依赖云端的问题,提出云边协同的渐进式推理系统PICE,通过动态调度、集成学习等提升吞吐量与质量,实验验证其吞吐量提升1.5-2倍、延迟最高降43%。
摘要:
大语言模型(LLM)虽在众多领域掀起了交互式AI应用的新浪潮,却存在推理成本高、严重依赖云端的问题。受语言学中的冗余现象启发,我们提出了一种云边协同的渐进式推理范式:即先由云端的LLM生成答案的梗概,再由边缘端的小模型(SLM)进行并行扩展以填充细节。渐进式推理有望提升吞吐量、降低推理延迟,但也面临关键的实现挑战,包括SLM导致的响应质量下降、梗概简洁性与全面性之间的权衡,以及网络传输和边缘推理带来的延迟增加。在本工作中,我们提出并实现了PICE——一个具备语义级云边协同能力的LLM服务系统,通过动态推理任务调度、集成学习和并行边缘推理来提升推理吞吐量与质量。大量测试床实验表明,我们的方法实现了1.5-2倍的吞吐量提升,延迟降低最高达43%,同时与SOTA系统相比还有望提升质量。
英文摘要:
Large language models (LLMs), while driving a new wave of interactive AI applications across numerous domains, suffer from high inference costs and heavy cloud dependency. Motivated by the redundancy phenomenon in linguistics, we propose a progressive inference paradigm over cloud and edge, i.e., firstly generating the sketch of the answer by LLMs at cloud, and then conducting parallel extension to fill in details by small models (SLMs) at edge. Progressive inference offers potential benefits to improve throughput and reduce inference latency while facing key implementation challenges, including decreased response quality from SLMs, a tradeoff between the brevity and comprehensiveness of sketches, as well as increased latency caused by network transmission and edge inference. In this work, we propose and implement PICE, an LLM serving system with semantic-level cloud-edge collaboration, enhancing inference throughput and quality through dynamic inference task scheduling, ensemble learning, and parallel edge inference. Extensive testbed experiments illustrate that our approach achieves $1.5-2\times$ throughput enhancement and up to 43% latency reduction, while also potentially enhancing the quality compared to SOTA systems.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。