Generating ultra-long sequences with large language models (LLMs) has become increasingly crucial but remains a highly time-intensive task, particularly for sequences up to 100K tokens. While traditional speculative decoding methods exist, simply extending their generation limits fails to accelerate the process and can be detrimental. Through an in-depth analysis, we identify three major challenges hindering efficient generation: frequent model reloading, dynamic key-value (KV) management and repetitive generation. To address these issues, we introduce TOKENSWIFT, a novel framework designed to substantially accelerate the generation process of ultra-long sequences while maintaining the target model's inherent quality. Experimental results demonstrate that TOKENSWIFT achieves over 3 times speedup across models of varying scales (1.5B, 7B, 8B, 14B) and architectures (MHA, GQA). This acceleration translates to hours of time savings for ultra-long sequence generation, establishing TOKENSWIFT as a scalable and effective solution at unprecedented lengths. Code can be found at https://github.com/bigai-nlco/TokenSwift.
Parallel Decoding via Hidden Transfer for Lossless Large Language Model Acceleration
Wangxuan Institute of Computer Technology, Peking University(北京大学王选计算机研究所);Center for Data Science, AAIS, Peking University(北京大学AAIS数据科学中心);Meituan(美团);Meta AI;National Key Laboratory of General Artificial Intelligence(通用人工智能全国重点实验室);BIGAI(北京智源人工智能研究院)
Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads
Princeton University(普林斯顿大学);Together AI;University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Carnegie Mellon University(卡内基梅隆大学);University of Connecticut(康涅狄格大学)
Self Speculative Decoding for Diffusion Large Language Models
School of AI, Shanghai Jiao Tong University(上海交通大学人工智能学院);University of Science and Technology of China(中国科学技术大学);Xidian University(西安电子科技大学);Shanghai AI Laboratory(上海人工智能实验室);Huawei Technologies Ltd.(华为技术有限公司)
SAM Decoding: Speculative Decoding via Suffix Automaton
School of Information, Renmin University of China(中国人民大学信息学院);Key Laboratory of Data Engineering and Knowledge Engineering(数据工程与知识工程重点实验室);Knowledge Engineering Group, Tsinghua University(清华大学知识工程组);Engineering Research Center of Database and Business Intelligence(数据库与商业智能工程研究中心)
Efficient Inference for Large Language Model-based Generative Recommendation
National University of Singapore(新加坡国立大学);Tsinghua University(清华大学);University of Science and Technology of China(中国科学技术大学);The Hong Kong Polytechnic University(香港理工大学)