arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.02532cs.CL
收藏

SpecExec:面向消费级设备上交互式 LLM 推理的大规模并行推测解码

SpecExec: Massively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devices

Carnegie Mellon University(卡内基梅隆大学);Meta AI;Together AI;Yandex;HSE University(高等经济大学)

Ruslan Svirschevski, Avner May, Zhuoming Chen, Beidi Chen, Zhihao Jia, Max Ryabinin

AI总结:

针对消费级设备难以本地运行 50B+ 参数 LLM 的问题,提出 SpecExec 并行推测解码,通过草稿概率构建缓存树并由目标模型一次验证,实现 RAM 卸载下每秒数 token 的交互推理。

摘要:

随着大语言模型被广泛采用,高效运行它们变得至关重要。近期关于 LLM 推理的研究使用推测解码来实现极高的加速。然而,这些研究大多隐含地将其算法设计面向高端数据中心硬件。在本工作中,我们提出相反的问题:在消费级机器上运行 LLM 能有多快?消费级 GPU 已无法容纳现有最大模型(50B+ 参数),必须将其卸载到 RAM 或 SSD。在使用卸载参数运行时,推理引擎可以像处理单个 token 一样同时处理数百或数千个 token 的批次,这使其天然适合推测解码。我们提出 SpecExec(Speculative Execution,推测执行),这是一种简单的并行解码方法,针对主流 LLM 系列,每次目标模型迭代最多可生成 20 个 token。它利用了现代 LLM 中 token 概率分布的高峰度特性,以及模型输出概率之间的高度一致性。SpecExec 从草稿模型中获取最可能的 token 续写,为目标模型构建一棵“缓存”树,随后在一次遍历中完成验证。使用 SpecExec,我们展示了在消费级 GPU 上通过 RAM 卸载推理 50B+ 参数 LLM:采用 4-bit 量化时速度为每秒 4–6 个 token,采用 16-bit 权重时速度为每秒 2–3 个 token。

英文摘要:

As large language models gain widespread adoption, running them efficiently becomes crucial. Recent works on LLM inference use speculative decoding to achieve extreme speedups. However, most of these works implicitly design their algorithms for high-end datacenter hardware. In this work, we ask the opposite question: how fast can we run LLMs on consumer machines? Consumer GPUs can no longer fit the largest available models (50B+ parameters) and must offload them to RAM or SSD. When running with offloaded parameters, the inference engine can process batches of hundreds or thousands of tokens at the same time as just one token, making it a natural fit for speculative decoding. We propose SpecExec (Speculative Execution), a simple parallel decoding method that can generate up to 20 tokens per target model iteration for popular LLM families. It utilizes the high spikiness of the token probabilities distribution in modern LLMs and a high degree of alignment between model output probabilities. SpecExec takes the most probable tokens continuation from the draft model to build a "cache" tree for the target model, which then gets validated in a single pass. Using SpecExec, we demonstrate inference of 50B+ parameter LLMs on consumer GPUs with RAM offloading at 4-6 tokens per second with 4-bit quantization or 2-3 tokens per second with 16-bit weights.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑