Speculative Decoding (SD) is a technique to accelerate the inference of Large Language Models (LLMs) by using a lower complexity draft model to propose candidate tokens verified by a larger target model. To further improve efficiency, Multi-Candidate Speculative Decoding (MCSD) improves upon this by sampling multiple candidate tokens from the draft model at each step and verifying them in parallel, thus increasing the chances of accepting a token and reducing generation time. Existing MCSD methods rely on the draft model to initialize the multi-candidate sequences and use static length and tree attention structure for draft generation. However, such an approach suffers from the draft and target model's output distribution differences, especially in a dynamic generation context. In this work, we introduce a new version of MCSD that includes a target model initialized multi-candidate generation, a dynamic sliced topology-aware causal mask for dynamic length adjustment, and decision models to optimize early stopping. We experimented with our method on Llama 2-7B and its variants and observed a maximum 27.5% speedup compared to our MCSD baseline across three benchmarks with Llama 2-7B as the target model and JackFram 68M as the draft model. Additionally, we evaluate the effects of using the target model initialized multi-candidate process with different draft models on output quality.
Ouroboros: Generating Longer Drafts Phrase by Phrase for Faster Speculative Decoding
Institute for Artificial Intelligence(人工智能研究院);Department of Computer Science and Technology(计算机科学与技术系);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);ModelBest Inc.(模佰科技);Beijing National Research Center for Information Science and Technology(北京信息科学与技术国家研究中心);Tsinghua University(清华大学)
GliDe with a CaPE: A Low-Hassle Method to Accelerate Speculative Decoding
Singapore Management University(新加坡管理大学);National University of Singapore(新加坡国立大学);The Hong Kong Polytechnic University (PolyU)(香港理工大学);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳));Tencent AI Lab(腾讯AI Lab)
AdaSD: Adaptive Speculative Decoding for Efficient Language Model Inference
Institute of Information Science, Academia Sinica(学术院信息研究所);Department of Computer Science and Information Engineering, National Taiwan University(台湾大学计算机科学与信息工程系)
FlexDraft: Flexible Speculative Decoding via Attention Tuning and Bonus-Guided Calibration
EPIC Lab, SJTU(上海交通大学EPIC实验室);UESTC;School of Software Engineering, HUST(华中科技大学软件工程学院);Tsinghua University(清华大学);HKUST(GZ)(香港科技大学(广州));Shanghai AI Laboratory(上海人工智能实验室)
Self Speculative Decoding for Diffusion Large Language Models
School of AI, Shanghai Jiao Tong University(上海交通大学人工智能学院);University of Science and Technology of China(中国科学技术大学);Xidian University(西安电子科技大学);Shanghai AI Laboratory(上海人工智能实验室);Huawei Technologies Ltd.(华为技术有限公司)
DREAM: Drafting with Refined Target Features and Entropy-Adaptive Cross-Attention Fusion for Multimodal Speculative Decoding
Courant Institute of Mathematical Sciences, New York University(纽约大学数学科学学院);Tandon School of Engineering, New York University(纽约大学工程学院);Cerebras Systems Inc.(Cerebras Systems公司);University of Pennsylvania(宾夕法尼亚大学)
AdaSpec: Adaptive Speculative Decoding for Fast, SLO-Aware Large Language Model Serving
Tongji University(同济大学);Shenzhen Research Institute of Big Data, The Chinese University of Hong Kong, Shenzhen(深圳大数据研究院,香港中文大学(深圳));Huazhong University of Science and Technology(华中科技大学);School of Data Science, The Chinese University of Hong Kong, Shenzhen(数据科学学院,香港中文大学(深圳))
Mistletoe: Stealthy Acceleration-Collapse Attacks on Speculative Decoding
Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));South China University of Technology(华南理工大学);Tsinghua Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);Huawei Technology(华为技术)
School of Artificial Intelligence, Wuhan University(武汉大学人工智能学院);School of Computer Science, Wuhan University(武汉大学计算机学院);School of Information Management, Wuhan University(武汉大学信息管理学院);School of Computer Science, Shanghai Jiao Tong University(上海交通大学计算机学院)