arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2502.02789cs.CLcs.AI
收藏

推测式预填充:通过轻量且免训练的token重要性估计为TTFT提速

Speculative Prefill: Turbocharging TTFT with Lightweight and Training-Free Token Importance Estimation

The University of Chicago(芝加哥大学);Carnegie Mellon University(卡内基梅隆大学)

Jingyu Liu, Beidi Chen, Ce Zhang

AI总结:

针对大语言模型推理首token生成时间(TTFT)优化难的问题,提出免训练框架SpecPrefill,通过轻量模型筛选重要提示token加速推理,实现最高7倍端到端QPS和7.66倍TTFT提升。

摘要:

缩短首token生成时间(time-to-first-token,TTFT)是现代大语言模型(large language model,LLM)推理引擎的核心重要目标。优化TTFT可直接提升最大每秒查询数(QPS),满足诸多关键应用的需求。然而,提升TTFT的难度极大,因为它受计算资源限制,且性能瓶颈已从以往诸多研究关注的自注意力模块转移到MLP部分。本文提出SpecPrefill——一种免训练框架,可加速长上下文和中上下文查询的推理TTFT,其核心洞察是:LLM的泛化能力足够强,仅输入精心挑选的提示token子集就能保证生成质量。SpecPrefill的核心是利用一个轻量模型,基于上下文推测局部重要的token,随后将这些token连同必要的位置信息送入主模型处理。我们在多样任务上评估了SpecPrefill,并在真实端到端场景和消融实验中对性能提升进行了全面基准测试。在真实下游任务上,SpecPrefill支撑Llama-3.1-405B-Instruct-FP8推理时,最大端到端QPS最高可达7倍,TTFT提升达7.66倍。

英文摘要:

Improving time-to-first-token (TTFT) is an essentially important objective in modern large language model (LLM) inference engines. Optimizing TTFT directly results in higher maximal QPS and meets the requirements of many critical applications. However, boosting TTFT is notoriously challenging since it is compute-bounded and the performance bottleneck shifts from the self-attention that many prior works focus on to the MLP part. In this work, we present SpecPrefill, a training free framework that accelerates the inference TTFT for both long and medium context queries based on the following insight: LLMs are generalized enough to preserve the quality given only a carefully chosen subset of prompt tokens. At its core, SpecPrefill leverages a lightweight model to speculate locally important tokens based on the context. These tokens, along with the necessary positional information, are then sent to the main model for processing. We evaluate SpecPrefill with a diverse set of tasks, followed by a comprehensive benchmarking of performance improvement both in a real end-to-end setting and ablation studies. SpecPrefill manages to serve Llama-3.1-405B-Instruct-FP8 with up to 7$\times$ maximal end-to-end QPS on real downstream tasks and 7.66$\times$ TTFT improvement.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑