T2D: Target-to-Draft Model Pruning & Adaptation for Efficient Speculative Decoding
Bellini College of AI, Cybersecurity, and Computing, University of South Florida(南佛罗里达大学贝利尼人工智能、网络安全与计算学院);School of Computing and Augmented Intelligence, Arizona State University(亚利桑那州立大学计算与增强智能学院);University of California, Davis(加州大学戴维斯分校)
Shahriar Kabir Nahin, Xiao Ye, Muhao Chen, Ben Zhou, Anshuman Chhabra
Speculative decoding significantly accelerates autoregressive text generation in Large Language Models (LLMs), but its efficacy depends largely on two key factors: the distributional alignment between the draft and the target model, and the draft model's latency during inference. However, existing approaches typically utilize independently pretrained small draft models or seek to prune the target model to a manually chosen architecture, thereby failing to explicitly balance the trade-off between draft model alignment and token generation efficiency. To address this gap, we formulate draft model design as an optimization problem and propose T2D, a framework that constructs draft models directly from the target model using latency-aware structured pruning and knowledge distillation. In this manner, T2D obtains draft models that are jointly optimized for both measured inference latency and cross-entropy loss on the target model's responses, resulting in superlative speculative decoding performance. Through extensive experiments on multi-domain speculative decoding benchmarks and across several target model families of differing scale, we demonstrate how T2D prunes up to 92\% of the target model to obtain highly performant draft models compared to competitive baselines. Moreover, draft models obtained via T2D can also be utilized by other speculative decoding token-alignment and knowledge distillation methods in a plug-and-play fashion, leading to further gains. Our findings underscore the importance of jointly navigating the draft model architecture-alignment frontier to unlock the full potential of speculative decoding for LLMs.
Speculative Decoding with CTC-based Draft Model for LLM Inference Acceleration
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);Key Laboratory of Al Safety, Chinese Academy of Sciences(中国科学院人工智能安全重点实验室)
When Drafts Evolve: Speculative Decoding Meets Online Learning
National Key Laboratory for Novel Software Technology, Nanjing University(南京大学新型软件技术国家重点实验室);School of Artificial Intelligence, Nanjing University(南京大学人工智能学院);University of California, San Diego(加州大学圣地亚哥分校)
AdaEAGLE: Optimizing Speculative Decoding via Explicit Modeling of Adaptive Draft Structures
X-LANCE Lab(X-LANCE实验室);Department of Computer Science and Engineering(计算机科学与工程系);MoE Key Lab of Artificial Intelligence(教育部人工智能重点实验室);SJTU AI Institute(上海交通大学人工智能研究院);Shanghai Jiao Tong University(上海交通大学)
HiViS: Hiding Visual Tokens from the Drafter for Speculative Decoding in Vision-Language Models
C 2 DL, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);AIRIA;City University of Hong Kong(香港城市大学)
TETRIS: Optimal Draft Token Selection for Batch Speculative Decoding
Singapore-MIT Alliance for Research and Technology(新加坡-麻省理工联合研究技术联盟);Dept. of Computer Science, National University of Singapore(新加坡国立大学计算机科学系);CSAIL, Massachusetts Institute of Technology(麻省理工学院计算机科学与人工智能实验室)
Mistletoe: Stealthy Acceleration-Collapse Attacks on Speculative Decoding
Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));South China University of Technology(华南理工大学);Tsinghua Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);Huawei Technology(华为技术)
AdaFlash: Adaptive Speculative Decoding via On-Policy Distilled Diffusion Drafters
State Key Laboratory for Novel Software Technology, Nanjing University(南京大学计算机软件新技术国家重点实验室);School of Artificial Intelligence, Nanjing University(南京大学人工智能学院);Huawei Foundation Model Dept(华为基础模型部)
Ouroboros: Generating Longer Drafts Phrase by Phrase for Faster Speculative Decoding
Institute for Artificial Intelligence(人工智能研究院);Department of Computer Science and Technology(计算机科学与技术系);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);ModelBest Inc.(模佰科技);Beijing National Research Center for Information Science and Technology(北京信息科学与技术国家研究中心);Tsinghua University(清华大学)
Draft-OPD: On-Policy Distillation for Speculative Draft Models
Shanghai Jiao Tong University(上海交通大学);Shanghai AI Laboratory(上海人工智能实验室);Tsinghua University(清华大学);The Chinese University of Hong Kong(香港中文大学);Peking University(北京大学);Zhejiang University(浙江大学)
PACER: Blockwise Pre-verification for Speculative Decoding with Adaptive Length
X-LANCE Lab, School of Computer Science MoE Key Lab of Artificial Intelligence, SJTU AI Institute Shanghai Jiao Tong University(X-LANCE实验室,计算机科学学院,人工智能关键实验室,上海交通大学人工智能研究院)