arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2502.08910cs.CLcs.LG
收藏

InfiniteHiP:在单GPU上将语言模型上下文扩展至300万token

InfiniteHiP: Extending Language Model Context Up to 3 Million Tokens on a Single GPU

KAIST(韩国科学技术院);Graduate School of AI, KAIST(韩国科学技术院人工智能学院)

Heejun Lee, Geon Park, Jaduk Suh, Sung Ju Hwang

AI总结:

针对LLM长上下文处理的速度慢、内存成本高及泛化能力不足问题,提出InfiniteHiP推理框架,通过分层token剪枝、选择性RoPE调整和键值缓存卸载,在单GPU上支持300万token上下文,实现18.95倍加速且无需额外训练。

摘要:

在现代大型语言模型(LLMs)中,处理极长的上下文长度面临重大挑战,会导致推理速度变慢和内存成本增加。此外,大多数现有预训练LLMs无法泛化到原始训练序列长度之外。为实现高效且实用的长上下文利用,我们提出InfiniteHiP,这是一种新颖且实用的LLM推理框架,通过模块化分层token剪枝算法动态消除无关上下文token来加速处理。我们的方法还通过根据LLMs内部注意力模式选择性应用多种RoPE调整方法,实现对更长序列的泛化。此外,我们将键值缓存卸载到主机内存,显著降低GPU内存压力。结果显示,InfiniteHiP在单个L40s 48GB GPU上可处理多达300万token——是原规模的3倍——且无上下文信息永久丢失。我们的框架在100万token上下文的注意力解码中实现18.95倍加速,无需额外训练。我们在SGLang框架中实现了该方法,并通过广泛评估证明其有效性和实用性。

英文摘要:

In modern large language models (LLMs), handling very long context lengths presents significant challenges as it causes slower inference speeds and increased memory costs. Additionally, most existing pre-trained LLMs fail to generalize beyond their original training sequence lengths. To enable efficient and practical long-context utilization, we introduce InfiniteHiP, a novel, and practical LLM inference framework that accelerates processing by dynamically eliminating irrelevant context tokens through a modular hierarchical token pruning algorithm. Our method also allows generalization to longer sequences by selectively applying various RoPE adjustment methods according to the internal attention patterns within LLMs. Furthermore, we offload the key-value cache to host memory during inference, significantly reducing GPU memory pressure. As a result, InfiniteHiP enables the processing of up to 3 million tokens on a single L40s 48GB GPU -- 3x larger -- without any permanent loss of context information. Our framework achieves an 18.95x speedup in attention decoding for a 1 million token context without requiring additional training. We implement our method in the SGLang framework and demonstrate its effectiveness and practicality through extensive evaluations.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑