Speculative Decoding is a widely used technique to speed up inference for Large Language Models (LLMs) without sacrificing quality. When performing inference, speculative decoding uses a smaller draft model to generate speculative tokens and then uses the target LLM to verify those draft tokens. The speedup provided by speculative decoding heavily depends on the choice of the draft model. In this work, we perform a detailed study comprising over 350 experiments with LLaMA-65B and OPT-66B using speculative decoding and delineate the factors that affect the performance gain provided by speculative decoding. Our experiments indicate that the performance of speculative decoding depends heavily on the latency of the draft model, and the draft model's capability in language modeling does not correlate strongly with its performance in speculative decoding. Based on these insights we explore a new design space for draft models and design hardware-efficient draft models for speculative decoding. Our newly designed draft model can provide 111% higher throughput than existing draft models and our approach generalizes further to all LLaMA models (1/2/3.1) and supervised fine-tuned models.
Mistletoe: Stealthy Acceleration-Collapse Attacks on Speculative Decoding
Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；South China University of Technology（华南理工大学）；Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Huawei Technology（华为技术）
Speculative Decoding with CTC-based Draft Model for LLM Inference Acceleration
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Key Laboratory of Al Safety, Chinese Academy of Sciences（中国科学院人工智能安全重点实验室）
AdaFlash: Adaptive Speculative Decoding via On-Policy Distilled Diffusion Drafters
State Key Laboratory for Novel Software Technology, Nanjing University（南京大学计算机软件新技术国家重点实验室）；School of Artificial Intelligence, Nanjing University（南京大学人工智能学院）；Huawei Foundation Model Dept（华为基础模型部）
Ouroboros: Generating Longer Drafts Phrase by Phrase for Faster Speculative Decoding
Institute for Artificial Intelligence（人工智能研究院）；Department of Computer Science and Technology（计算机科学与技术系）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；ModelBest Inc.（模佰科技）；Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）；Tsinghua University（清华大学）
Draft-OPD: On-Policy Distillation for Speculative Draft Models
Shanghai Jiao Tong University（上海交通大学）；Shanghai AI Laboratory（上海人工智能实验室）；Tsinghua University（清华大学）；The Chinese University of Hong Kong（香港中文大学）；Peking University（北京大学）；Zhejiang University（浙江大学）
FlexDraft: Flexible Speculative Decoding via Attention Tuning and Bonus-Guided Calibration
EPIC Lab, SJTU（上海交通大学EPIC实验室）；UESTC；School of Software Engineering, HUST（华中科技大学软件工程学院）；Tsinghua University（清华大学）；HKUST(GZ)（香港科技大学（广州））；Shanghai AI Laboratory（上海人工智能实验室）
DREAM: Drafting with Refined Target Features and Entropy-Adaptive Cross-Attention Fusion for Multimodal Speculative Decoding
Courant Institute of Mathematical Sciences, New York University（纽约大学数学科学学院）；Tandon School of Engineering, New York University（纽约大学工程学院）；Cerebras Systems Inc.（Cerebras Systems公司）；University of Pennsylvania（宾夕法尼亚大学）
DeLS-Spec: Decoupled Long-Short Contexts for Parallel Speculative Drafting
College of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics（南京航空航天大学人工智能学院）；MIIT Key Laboratory of Pattern Analysis and Machine Intelligence（工业和信息化部模式分析与机器智能重点实验室）；The Key Laboratory of Brain-Machine Intelligence Technology, Ministry of Education（教育部脑机智能技术重点实验室）
