As the usage of large language models (LLMs) grows, performing efficient inference with these models becomes increasingly important. While speculative decoding has recently emerged as a promising direction for speeding up inference, existing methods are limited in their ability to scale to larger speculation budgets, and adapt to different hyperparameters and hardware. This paper introduces Sequoia, a scalable, robust, and hardware-aware algorithm for speculative decoding. To attain better scalability, Sequoia introduces a dynamic programming algorithm to find the optimal tree structure for the speculated tokens. To achieve robust speculative performance, Sequoia uses a novel sampling and verification method that outperforms prior work across different decoding temperatures. Finally, Sequoia introduces a hardware-aware tree optimizer that maximizes speculative performance by automatically selecting the token tree size and depth for a given hardware platform. Evaluation shows that Sequoia improves the decoding speed of Llama2-7B, Llama2-13B, and Vicuna-33B on an A100 by up to $4.04\times$, $3.73\times$, and $2.27\times$. For offloading setting on L40, Sequoia achieves as low as 0.56 s/token for exact Llama2-70B inference latency, which is $9.96\times$ on our optimized offloading system (5.6 s/token), $9.7\times$ than DeepSpeed-Zero-Inference, $19.5\times$ than Huggingface Accelerate.
Ouroboros: Generating Longer Drafts Phrase by Phrase for Faster Speculative Decoding
Institute for Artificial Intelligence(人工智能研究院);Department of Computer Science and Technology(计算机科学与技术系);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);ModelBest Inc.(模佰科技);Beijing National Research Center for Information Science and Technology(北京信息科学与技术国家研究中心);Tsinghua University(清华大学)
GliDe with a CaPE: A Low-Hassle Method to Accelerate Speculative Decoding
Singapore Management University(新加坡管理大学);National University of Singapore(新加坡国立大学);The Hong Kong Polytechnic University (PolyU)(香港理工大学);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳));Tencent AI Lab(腾讯AI Lab)
AdaSD: Adaptive Speculative Decoding for Efficient Language Model Inference
Institute of Information Science, Academia Sinica(学术院信息研究所);Department of Computer Science and Information Engineering, National Taiwan University(台湾大学计算机科学与信息工程系)
FlexDraft: Flexible Speculative Decoding via Attention Tuning and Bonus-Guided Calibration
EPIC Lab, SJTU(上海交通大学EPIC实验室);UESTC;School of Software Engineering, HUST(华中科技大学软件工程学院);Tsinghua University(清华大学);HKUST(GZ)(香港科技大学(广州));Shanghai AI Laboratory(上海人工智能实验室)
Self Speculative Decoding for Diffusion Large Language Models
School of AI, Shanghai Jiao Tong University(上海交通大学人工智能学院);University of Science and Technology of China(中国科学技术大学);Xidian University(西安电子科技大学);Shanghai AI Laboratory(上海人工智能实验室);Huawei Technologies Ltd.(华为技术有限公司)
DREAM: Drafting with Refined Target Features and Entropy-Adaptive Cross-Attention Fusion for Multimodal Speculative Decoding
Courant Institute of Mathematical Sciences, New York University(纽约大学数学科学学院);Tandon School of Engineering, New York University(纽约大学工程学院);Cerebras Systems Inc.(Cerebras Systems公司);University of Pennsylvania(宾夕法尼亚大学)
AdaSpec: Adaptive Speculative Decoding for Fast, SLO-Aware Large Language Model Serving
Tongji University(同济大学);Shenzhen Research Institute of Big Data, The Chinese University of Hong Kong, Shenzhen(深圳大数据研究院,香港中文大学(深圳));Huazhong University of Science and Technology(华中科技大学);School of Data Science, The Chinese University of Hong Kong, Shenzhen(数据科学学院,香港中文大学(深圳))
Mistletoe: Stealthy Acceleration-Collapse Attacks on Speculative Decoding
Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));South China University of Technology(华南理工大学);Tsinghua Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);Huawei Technology(华为技术)
School of Artificial Intelligence, Wuhan University(武汉大学人工智能学院);School of Computer Science, Wuhan University(武汉大学计算机学院);School of Information Management, Wuhan University(武汉大学信息管理学院);School of Computer Science, Shanghai Jiao Tong University(上海交通大学计算机学院)