Wangxuan Institute of Computer Technology, Peking University(北京大学王选计算机研究所);Meituan(美团);Tianjin University(天津大学);National Key Laboratory of General Artificial Intelligence(通用人工智能全国重点实验室);Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院)
Speculative decoding has emerged as a promising technique to accelerate the inference of Large Language Models (LLMs) by employing a small language model to draft a hypothesis sequence, which is then validated by the LLM. The effectiveness of this approach heavily relies on the balance between performance and efficiency of the draft model. In our research, we focus on enhancing the proportion of draft tokens that are accepted to the final output by generating multiple hypotheses instead of just one. This allows the LLM more options to choose from and select the longest sequence that meets its standards. Our analysis reveals that hypotheses produced by the draft model share many common token sequences, suggesting a potential for optimizing computation. Leveraging this observation, we introduce an innovative approach utilizing a directed acyclic graph (DAG) to manage the drafted hypotheses. This structure enables us to efficiently predict and merge recurring token sequences, vastly reducing the computational demands of the draft model. We term this approach Graph-structured Speculative Decoding (GSD). We apply GSD across a range of LLMs, including a 70-billion parameter LLaMA-2 model, and observe a remarkable speedup of 1.73$\times$ to 1.96$\times$, significantly surpassing standard speculative decoding.
Ouroboros: Generating Longer Drafts Phrase by Phrase for Faster Speculative Decoding
Institute for Artificial Intelligence(人工智能研究院);Department of Computer Science and Technology(计算机科学与技术系);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);ModelBest Inc.(模佰科技);Beijing National Research Center for Information Science and Technology(北京信息科学与技术国家研究中心);Tsinghua University(清华大学)
GliDe with a CaPE: A Low-Hassle Method to Accelerate Speculative Decoding
Singapore Management University(新加坡管理大学);National University of Singapore(新加坡国立大学);The Hong Kong Polytechnic University (PolyU)(香港理工大学);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳));Tencent AI Lab(腾讯AI Lab)
AdaSD: Adaptive Speculative Decoding for Efficient Language Model Inference
Institute of Information Science, Academia Sinica(学术院信息研究所);Department of Computer Science and Information Engineering, National Taiwan University(台湾大学计算机科学与信息工程系)
FlexDraft: Flexible Speculative Decoding via Attention Tuning and Bonus-Guided Calibration
EPIC Lab, SJTU(上海交通大学EPIC实验室);UESTC;School of Software Engineering, HUST(华中科技大学软件工程学院);Tsinghua University(清华大学);HKUST(GZ)(香港科技大学(广州));Shanghai AI Laboratory(上海人工智能实验室)
Self Speculative Decoding for Diffusion Large Language Models
School of AI, Shanghai Jiao Tong University(上海交通大学人工智能学院);University of Science and Technology of China(中国科学技术大学);Xidian University(西安电子科技大学);Shanghai AI Laboratory(上海人工智能实验室);Huawei Technologies Ltd.(华为技术有限公司)
DREAM: Drafting with Refined Target Features and Entropy-Adaptive Cross-Attention Fusion for Multimodal Speculative Decoding
Courant Institute of Mathematical Sciences, New York University(纽约大学数学科学学院);Tandon School of Engineering, New York University(纽约大学工程学院);Cerebras Systems Inc.(Cerebras Systems公司);University of Pennsylvania(宾夕法尼亚大学)
AdaSpec: Adaptive Speculative Decoding for Fast, SLO-Aware Large Language Model Serving
Tongji University(同济大学);Shenzhen Research Institute of Big Data, The Chinese University of Hong Kong, Shenzhen(深圳大数据研究院,香港中文大学(深圳));Huazhong University of Science and Technology(华中科技大学);School of Data Science, The Chinese University of Hong Kong, Shenzhen(数据科学学院,香港中文大学(深圳))
Mistletoe: Stealthy Acceleration-Collapse Attacks on Speculative Decoding
Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));South China University of Technology(华南理工大学);Tsinghua Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);Huawei Technology(华为技术)
School of Artificial Intelligence, Wuhan University(武汉大学人工智能学院);School of Computer Science, Wuhan University(武汉大学计算机学院);School of Information Management, Wuhan University(武汉大学信息管理学院);School of Computer Science, Shanghai Jiao Tong University(上海交通大学计算机学院)