Text generation with Large Language Models (LLMs) is known to be memory bound due to the combination of their auto-regressive nature, huge parameter counts, and limited memory bandwidths, often resulting in low token rates. Speculative decoding has been proposed as a solution for LLM inference acceleration. However, since draft models are often unavailable in the modern open-source LLM families, e.g., for Llama 2 7B, training a high-quality draft model is required to enable inference acceleration via speculative decoding. In this paper, we propose a simple draft model training framework for direct alignment to chat-capable target models. With the proposed framework, we train Llama 2 Chat Drafter 115M, a draft model for Llama 2 Chat 7B or larger, with only 1.64\% of the original size. Our training framework only consists of pretraining, distillation dataset generation, and finetuning with knowledge distillation, with no additional alignment procedure. For the finetuning step, we use instruction-response pairs generated by target model for distillation in plausible data distribution, and propose a new Total Variation Distance++ (TVD++) loss that incorporates variance reduction techniques inspired from the policy gradient method in reinforcement learning. Our empirical results show that Llama 2 Chat Drafter 115M with speculative decoding achieves up to 2.3 block efficiency and 2.4$\times$ speed-up relative to autoregressive decoding on various tasks with no further task-specific fine-tuning.
Speculative Decoding with CTC-based Draft Model for LLM Inference Acceleration
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);Key Laboratory of Al Safety, Chinese Academy of Sciences(中国科学院人工智能安全重点实验室)
When Drafts Evolve: Speculative Decoding Meets Online Learning
National Key Laboratory for Novel Software Technology, Nanjing University(南京大学新型软件技术国家重点实验室);School of Artificial Intelligence, Nanjing University(南京大学人工智能学院);University of California, San Diego(加州大学圣地亚哥分校)
AdaEAGLE: Optimizing Speculative Decoding via Explicit Modeling of Adaptive Draft Structures
X-LANCE Lab(X-LANCE实验室);Department of Computer Science and Engineering(计算机科学与工程系);MoE Key Lab of Artificial Intelligence(教育部人工智能重点实验室);SJTU AI Institute(上海交通大学人工智能研究院);Shanghai Jiao Tong University(上海交通大学)
HiViS: Hiding Visual Tokens from the Drafter for Speculative Decoding in Vision-Language Models
C 2 DL, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);AIRIA;City University of Hong Kong(香港城市大学)
TETRIS: Optimal Draft Token Selection for Batch Speculative Decoding
Singapore-MIT Alliance for Research and Technology(新加坡-麻省理工联合研究技术联盟);Dept. of Computer Science, National University of Singapore(新加坡国立大学计算机科学系);CSAIL, Massachusetts Institute of Technology(麻省理工学院计算机科学与人工智能实验室)
Mistletoe: Stealthy Acceleration-Collapse Attacks on Speculative Decoding
Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));South China University of Technology(华南理工大学);Tsinghua Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);Huawei Technology(华为技术)
Ouroboros: Generating Longer Drafts Phrase by Phrase for Faster Speculative Decoding
Institute for Artificial Intelligence(人工智能研究院);Department of Computer Science and Technology(计算机科学与技术系);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);ModelBest Inc.(模佰科技);Beijing National Research Center for Information Science and Technology(北京信息科学与技术国家研究中心);Tsinghua University(清华大学)
AdaFlash: Adaptive Speculative Decoding via On-Policy Distilled Diffusion Drafters
State Key Laboratory for Novel Software Technology, Nanjing University(南京大学计算机软件新技术国家重点实验室);School of Artificial Intelligence, Nanjing University(南京大学人工智能学院);Huawei Foundation Model Dept(华为基础模型部)
Draft-OPD: On-Policy Distillation for Speculative Draft Models
Shanghai Jiao Tong University(上海交通大学);Shanghai AI Laboratory(上海人工智能实验室);Tsinghua University(清华大学);The Chinese University of Hong Kong(香港中文大学);Peking University(北京大学);Zhejiang University(浙江大学)
PACER: Blockwise Pre-verification for Speculative Decoding with Adaptive Length
X-LANCE Lab, School of Computer Science MoE Key Lab of Artificial Intelligence, SJTU AI Institute Shanghai Jiao Tong University(X-LANCE实验室,计算机科学学院,人工智能关键实验室,上海交通大学人工智能研究院)