缩短首token生成时间(time-to-first-token,TTFT)是现代大语言模型(large language model,LLM)推理引擎的核心重要目标。优化TTFT可直接提升最大每秒查询数(QPS),满足诸多关键应用的需求。然而,提升TTFT的难度极大,因为它受计算资源限制,且性能瓶颈已从以往诸多研究关注的自注意力模块转移到MLP部分。本文提出SpecPrefill——一种免训练框架,可加速长上下文和中上下文查询的推理TTFT,其核心洞察是:LLM的泛化能力足够强,仅输入精心挑选的提示token子集就能保证生成质量。SpecPrefill的核心是利用一个轻量模型,基于上下文推测局部重要的token,随后将这些token连同必要的位置信息送入主模型处理。我们在多样任务上评估了SpecPrefill,并在真实端到端场景和消融实验中对性能提升进行了全面基准测试。在真实下游任务上,SpecPrefill支撑Llama-3.1-405B-Instruct-FP8推理时,最大端到端QPS最高可达7倍,TTFT提升达7.66倍。
英文摘要:
Improving time-to-first-token (TTFT) is an essentially important objective in modern large language model (LLM) inference engines. Optimizing TTFT directly results in higher maximal QPS and meets the requirements of many critical applications. However, boosting TTFT is notoriously challenging since it is compute-bounded and the performance bottleneck shifts from the self-attention that many prior works focus on to the MLP part. In this work, we present SpecPrefill, a training free framework that accelerates the inference TTFT for both long and medium context queries based on the following insight: LLMs are generalized enough to preserve the quality given only a carefully chosen subset of prompt tokens. At its core, SpecPrefill leverages a lightweight model to speculate locally important tokens based on the context. These tokens, along with the necessary positional information, are then sent to the main model for processing. We evaluate SpecPrefill with a diverse set of tasks, followed by a comprehensive benchmarking of performance improvement both in a real end-to-end setting and ablation studies. SpecPrefill manages to serve Llama-3.1-405B-Instruct-FP8 with up to 7$\times$ maximal end-to-end QPS on real downstream tasks and 7.66$\times$ TTFT improvement.
RelayCaching: Accelerating LLM Collaboration via Decoding KV Cache Reuse
Beijing University of Posts(北京邮电大学);Tsinghua University, Beijing, China(清华大学);University of Electronic Science(电子科学大学);Peking University, Beijing, China(北京大学)
Prefill/Decode-Aware Evaluation of LLM Inference on Emerging AI Accelerators
Department of Computer Science(计算机科学系);San Francisco State University(旧金山州立大学);Argonne National Laboratory(阿贡国家实验室);Lawrence Berkeley National Laboratory(伯克利国家实验室)
School of Information, Renmin University of China(中国人民大学信息学院);Key Laboratory of Data Engineering and Knowledge Engineering, Beijing, China(数据工程与知识工程重点实验室);Engineering Research Center of Database and Business Intelligence, Beijing, China(数据库与商业智能工程研究中心);Ant Group, Hangzhou, China(蚂蚁集团)
Just Let Linear States Forget the Distant Past: Prefix Caching via Suffix Replay for Hybrid LLMs
Institute of Artificial Intelligence, China Telecom (TeleAI)(中国电信(TeleAI)人工智能研究所);Shanghai Jiao Tong University(上海交通大学);Tsinghua University(清华大学);Individual Researcher(独立研究者)