arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.12374cs.CL
收藏

Sequoia:可扩展、鲁棒且硬件感知的投机解码

Sequoia: Scalable, Robust, and Hardware-aware Speculative Decoding

Carnegie Mellon University(卡内基梅隆大学);Together AI;Yandex;National Research University Higher School of Economics(俄罗斯国家研究大学高等经济学院);FAIR, Meta(FAIR,Meta)

Zhuoming Chen, Avner May, Ruslan Svirschevski, Yuhsun Huang, Max Ryabinin, Zhihao Jia, Beidi Chen

AI总结:

本文提出 Sequoia 投机解码算法,通过动态规划优化 token 树结构、鲁棒采样验证和硬件感知树优化,在多种模型和硬件上显著提升了解码速度。

摘要:

随着大语言模型(LLMs)使用量的增长,对这些模型进行高效推理变得愈发重要。虽然投机解码(speculative decoding)近年来已成为加速推理的一个有前景的方向,但现有方法在扩展到更大的投机预算、适应不同超参数和硬件方面能力有限。本文提出 Sequoia,一种可扩展、鲁棒且硬件感知的投机解码算法。为了获得更好的可扩展性,Sequoia 引入了一种动态规划算法,为被推测的 token 寻找最优树结构。为了实现鲁棒的投机性能,Sequoia 使用了一种新颖的采样与验证方法,在不同解码温度下均优于先前工作。最后,Sequoia 引入了一个硬件感知的树优化器,通过针对给定硬件平台自动选择 token 树的大小和深度来最大化投机性能。评估表明,Sequoia 在 A100 上将 Llama2-7B、Llama2-13B 和 Vicuna-33B 的解码速度分别提升了最高 4.04 倍、3.73 倍和 2.27 倍。在 L40 上的卸载设置中,Sequoia 为精确的 Llama2-70B 推理延迟实现了低至 0.56 秒/token,这在我们优化的卸载系统(5.6 秒/token)上是 9.96 倍,比 DeepSpeed-Zero-Inference 快 9.7 倍,比 Huggingface Accelerate 快 19.5 倍。

英文摘要:

As the usage of large language models (LLMs) grows, performing efficient inference with these models becomes increasingly important. While speculative decoding has recently emerged as a promising direction for speeding up inference, existing methods are limited in their ability to scale to larger speculation budgets, and adapt to different hyperparameters and hardware. This paper introduces Sequoia, a scalable, robust, and hardware-aware algorithm for speculative decoding. To attain better scalability, Sequoia introduces a dynamic programming algorithm to find the optimal tree structure for the speculated tokens. To achieve robust speculative performance, Sequoia uses a novel sampling and verification method that outperforms prior work across different decoding temperatures. Finally, Sequoia introduces a hardware-aware tree optimizer that maximizes speculative performance by automatically selecting the token tree size and depth for a given hardware platform. Evaluation shows that Sequoia improves the decoding speed of Llama2-7B, Llama2-13B, and Vicuna-33B on an A100 by up to $4.04\times$, $3.73\times$, and $2.27\times$. For offloading setting on L40, Sequoia achieves as low as 0.56 s/token for exact Llama2-70B inference latency, which is $9.96\times$ on our optimized offloading system (5.6 s/token), $9.7\times$ than DeepSpeed-Zero-Inference, $19.5\times$ than Huggingface Accelerate.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑