Large language models have shown impressive capabilities across a variety of NLP tasks, yet their generating text autoregressively is time-consuming. One way to speed them up is speculative decoding, which generates candidate segments (a sequence of tokens) from a fast draft model that is then verified in parallel by the target model. However, the acceptance rate of candidate tokens receives limitations from several factors, such as the model, the dataset, and the decoding setup. This paper proposes sampling multiple candidates from a draft model and then organising them in batches for verification. We design algorithms for efficient multi-candidate verification while maintaining the distribution of the target model. Our approach shows significant improvements in acceptance rates on multiple datasets and models, consistently outperforming standard speculative decoding.
Ouroboros: Generating Longer Drafts Phrase by Phrase for Faster Speculative Decoding
Institute for Artificial Intelligence（人工智能研究院）；Department of Computer Science and Technology（计算机科学与技术系）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；ModelBest Inc.（模佰科技）；Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）；Tsinghua University（清华大学）
GliDe with a CaPE: A Low-Hassle Method to Accelerate Speculative Decoding
Singapore Management University（新加坡管理大学）；National University of Singapore（新加坡国立大学）；The Hong Kong Polytechnic University (PolyU)（香港理工大学）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））；Tencent AI Lab（腾讯AI Lab）
AdaSD: Adaptive Speculative Decoding for Efficient Language Model Inference
Institute of Information Science, Academia Sinica（学术院信息研究所）；Department of Computer Science and Information Engineering, National Taiwan University（台湾大学计算机科学与信息工程系）
FlexDraft: Flexible Speculative Decoding via Attention Tuning and Bonus-Guided Calibration
EPIC Lab, SJTU（上海交通大学EPIC实验室）；UESTC；School of Software Engineering, HUST（华中科技大学软件工程学院）；Tsinghua University（清华大学）；HKUST(GZ)（香港科技大学（广州））；Shanghai AI Laboratory（上海人工智能实验室）
Self Speculative Decoding for Diffusion Large Language Models
School of AI, Shanghai Jiao Tong University（上海交通大学人工智能学院）；University of Science and Technology of China（中国科学技术大学）；Xidian University（西安电子科技大学）；Shanghai AI Laboratory（上海人工智能实验室）；Huawei Technologies Ltd.（华为技术有限公司）
DREAM: Drafting with Refined Target Features and Entropy-Adaptive Cross-Attention Fusion for Multimodal Speculative Decoding
Courant Institute of Mathematical Sciences, New York University（纽约大学数学科学学院）；Tandon School of Engineering, New York University（纽约大学工程学院）；Cerebras Systems Inc.（Cerebras Systems公司）；University of Pennsylvania（宾夕法尼亚大学）
AdaSpec: Adaptive Speculative Decoding for Fast, SLO-Aware Large Language Model Serving
Tongji University（同济大学）；Shenzhen Research Institute of Big Data, The Chinese University of Hong Kong, Shenzhen（深圳大数据研究院，香港中文大学（深圳））；Huazhong University of Science and Technology（华中科技大学）；School of Data Science, The Chinese University of Hong Kong, Shenzhen（数据科学学院，香港中文大学（深圳））
Mistletoe: Stealthy Acceleration-Collapse Attacks on Speculative Decoding
Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；South China University of Technology（华南理工大学）；Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Huawei Technology（华为技术）
School of Artificial Intelligence, Wuhan University（武汉大学人工智能学院）；School of Computer Science, Wuhan University（武汉大学计算机学院）；School of Information Management, Wuhan University（武汉大学信息管理学院）；School of Computer Science, Shanghai Jiao Tong University（上海交通大学计算机学院）
TETRIS: Optimal Draft Token Selection for Batch Speculative Decoding
Singapore-MIT Alliance for Research and Technology（新加坡-麻省理工联合研究技术联盟）；Dept. of Computer Science, National University of Singapore（新加坡国立大学计算机科学系）；CSAIL, Massachusetts Institute of Technology（麻省理工学院计算机科学与人工智能实验室）
