Large Language Models (LLMs) exhibit high inference latency due to their autoregressive decoding nature. While the draft head in speculative decoding mitigates this issue, its full potential remains unexplored. In this paper, we introduce KOALA (K-layer Optimized Adversarial Learning Architecture), an orthogonal approach to the draft head. By transforming the conventional single-layer draft head into a multi-layer architecture and incorporating adversarial learning into the traditional supervised training, KOALA significantly improves the accuracy of the draft head in predicting subsequent tokens, thus more closely mirroring the functionality of LLMs. Although this improvement comes at the cost of slightly increased drafting overhead, KOALA substantially unlocks the draft head's potential, greatly enhancing speculative decoding. We conducted comprehensive evaluations of KOALA, including both autoregressive and non-autoregressive draft heads across various tasks, demonstrating a latency speedup ratio improvement of 0.24x-0.41x, which is 10.57%-14.09% faster than the original draft heads.
Ouroboros: Generating Longer Drafts Phrase by Phrase for Faster Speculative Decoding
Institute for Artificial Intelligence(人工智能研究院);Department of Computer Science and Technology(计算机科学与技术系);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);ModelBest Inc.(模佰科技);Beijing National Research Center for Information Science and Technology(北京信息科学与技术国家研究中心);Tsinghua University(清华大学)
Draft-OPD: On-Policy Distillation for Speculative Draft Models
Shanghai Jiao Tong University(上海交通大学);Shanghai AI Laboratory(上海人工智能实验室);Tsinghua University(清华大学);The Chinese University of Hong Kong(香港中文大学);Peking University(北京大学);Zhejiang University(浙江大学)
FlexDraft: Flexible Speculative Decoding via Attention Tuning and Bonus-Guided Calibration
EPIC Lab, SJTU(上海交通大学EPIC实验室);UESTC;School of Software Engineering, HUST(华中科技大学软件工程学院);Tsinghua University(清华大学);HKUST(GZ)(香港科技大学(广州));Shanghai AI Laboratory(上海人工智能实验室)
DREAM: Drafting with Refined Target Features and Entropy-Adaptive Cross-Attention Fusion for Multimodal Speculative Decoding
Courant Institute of Mathematical Sciences, New York University(纽约大学数学科学学院);Tandon School of Engineering, New York University(纽约大学工程学院);Cerebras Systems Inc.(Cerebras Systems公司);University of Pennsylvania(宾夕法尼亚大学)
DeLS-Spec: Decoupled Long-Short Contexts for Parallel Speculative Drafting
College of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics(南京航空航天大学人工智能学院);MIIT Key Laboratory of Pattern Analysis and Machine Intelligence(工业和信息化部模式分析与机器智能重点实验室);The Key Laboratory of Brain-Machine Intelligence Technology, Ministry of Education(教育部脑机智能技术重点实验室)
Learning to Draft: Adaptive Speculative Decoding with Reinforcement Learning
National Key Laboratory for Multimedia Information Processing, Peking University(多媒体信息处理国家级重点实验室,北京大学);School of Computer Science, Peking University(计算机科学学院,北京大学);Microsoft Research Asia(微软亚洲研究院)
GliDe with a CaPE: A Low-Hassle Method to Accelerate Speculative Decoding
Singapore Management University(新加坡管理大学);National University of Singapore(新加坡国立大学);The Hong Kong Polytechnic University (PolyU)(香港理工大学);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳));Tencent AI Lab(腾讯AI Lab)
Wangxuan Institute of Computer Technology, Peking University(北京大学王选计算机研究所);Meituan(美团);Tianjin University(天津大学);National Key Laboratory of General Artificial Intelligence(通用人工智能全国重点实验室);Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院)
DFlow: Enabling Verifier Information Flow in Block Diffusion Speculative Decoding
Peking University(北京大学);University of Electronic Science and Technology of China(电子科技大学);Shanghai Jiao Tong University(上海交通大学);Institute of Computational Social Science, Peking University (Qingdao)(北京大学(青岛)计算社会科学研究所)
Flatter Tokens are More Valuable for Speculative Draft Model Training
Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education(新一代人工智能技术及其交叉应用关键实验室(东南大学));Southeast University(东南大学);Nanjing University of Information Science and Technology(南京信息工程大学);Qiyuan Tech(启元科技)
Speculative Decoding with CTC-based Draft Model for LLM Inference Acceleration
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);Key Laboratory of Al Safety, Chinese Academy of Sciences(中国科学院人工智能安全重点实验室)
AdaEAGLE: Optimizing Speculative Decoding via Explicit Modeling of Adaptive Draft Structures
X-LANCE Lab(X-LANCE实验室);Department of Computer Science and Engineering(计算机科学与工程系);MoE Key Lab of Artificial Intelligence(教育部人工智能重点实验室);SJTU AI Institute(上海交通大学人工智能研究院);Shanghai Jiao Tong University(上海交通大学)
TETRIS: Optimal Draft Token Selection for Batch Speculative Decoding
Singapore-MIT Alliance for Research and Technology(新加坡-麻省理工联合研究技术联盟);Dept. of Computer Science, National University of Singapore(新加坡国立大学计算机科学系);CSAIL, Massachusetts Institute of Technology(麻省理工学院计算机科学与人工智能实验室)