arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2502.18890cs.CL
收藏

TokenSwift:超长序列生成的无损加速

TokenSwift: Lossless Acceleration of Ultra Long Sequence Generation

NLCo Lab, BIGAI(NLCo实验室,BIGAI);LUMIA Lab, Shanghai Jiao Tong University(LUMIA实验室,上海交通大学)

Tong Wu, Junzhe Shen, Zixia Jia, Yuxuan Wang, Zilong Zheng

AI总结:

针对超长序列生成耗时问题,提出TOKENSWIFT框架,通过解决模型重载、KV管理和重复生成三大挑战,在多种模型上实现超3倍加速,且不损失质量。

摘要:

使用大型语言模型(LLM)生成超长序列已变得日益关键,但这一过程仍然高度耗时,尤其是对于长达10万令牌的序列。虽然存在传统的推测解码方法,但简单地扩展其生成限制并不能加速该过程,反而可能产生不利影响。通过深入分析,我们识别出阻碍高效生成的三大主要挑战:频繁的模型重载、动态键值(KV)管理和重复生成。为解决这些问题,我们引入了TOKENSWIFT,一种新颖的框架,旨在大幅加速超长序列的生成过程,同时保持目标模型的固有质量。实验结果表明,TOKENSWIFT在不同规模(1.5B、7B、8B、14B)和不同架构(MHA、GQA)的模型上实现了超过3倍的加速。这一加速为超长序列生成节省了数小时的时间,使TOKENSWIFT成为在空前长度下可扩展且有效的解决方案。代码可在https://github.com/bigai-nlco/TokenSwift获取。

英文摘要:

Generating ultra-long sequences with large language models (LLMs) has become increasingly crucial but remains a highly time-intensive task, particularly for sequences up to 100K tokens. While traditional speculative decoding methods exist, simply extending their generation limits fails to accelerate the process and can be detrimental. Through an in-depth analysis, we identify three major challenges hindering efficient generation: frequent model reloading, dynamic key-value (KV) management and repetitive generation. To address these issues, we introduce TOKENSWIFT, a novel framework designed to substantially accelerate the generation process of ultra-long sequences while maintaining the target model's inherent quality. Experimental results demonstrate that TOKENSWIFT achieves over 3 times speedup across models of varying scales (1.5B, 7B, 8B, 14B) and architectures (MHA, GQA). This acceleration translates to hours of time savings for ultra-long sequence generation, establishing TOKENSWIFT as a scalable and effective solution at unprecedented lengths. Code can be found at https://github.com/bigai-nlco/TokenSwift.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑