arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.01869cs.LGcs.CLcs.DC
收藏

InferCept:面向增强型大型语言模型推理的高效拦截支持

InferCept: Efficient Intercept Support for Augmented Large Language Model Inference

University of California, San Diego(加利福尼亚大学圣迭戈分校)

Reyna Abhyankar, Zijian He, Vikranth Srivatsa, Hao Zhang, Yiying Zhang

AI总结:

本文提出InferCept框架,通过高效拦截机制解决增强型LLM推理中外部交互导致的重复计算问题,节省GPU资源并提升内存利用率,使整体吞吐量提升1.6-2倍。

摘要:

大型语言模型正日益与外部环境、工具和智能体(如ChatGPT插件)集成,以将其能力扩展到以语言为中心的任务之外。然而,当前的LLM推理系统是为独立LLM设计的。它们将每次外部交互视为LLM生成的结束,并在交互完成时形成新的请求,导致对已计算上下文的不必要重计算,这占用了模型总前向传播时间的37-40%。本文提出了InferCept,这是首个针对增强型LLM的LLM推理框架,支持对LLM生成的高效拦截。InferCept最小化了由LLM拦截引起的GPU资源浪费,并将节省的内存专用于服务更多请求。与最先进的LLM推理系统相比,InferCept将整体服务吞吐量提高了1.6倍至2倍,并且每秒完成的请求数量增加了2倍。

英文摘要:

Large language models are increasingly integrated with external environments, tools, and agents like ChatGPT plugins to extend their capability beyond language-centric tasks. However, today's LLM inference systems are designed for standalone LLMs. They treat each external interaction as the end of LLM generation and form a new request when the interaction finishes, causing unnecessary recomputation of already computed contexts, which accounts for 37-40% of total model forwarding time. This paper presents InferCept, the first LLM inference framework targeting augmented LLMs and supporting the efficient interception of LLM generation. InferCept minimizes the GPU resource waste caused by LLM interceptions and dedicates saved memory for serving more requests. InferCept improves the overall serving throughput by 1.6x-2x and completes 2x more requests per second compared to the state-of-the-art LLM inference systems.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑