发现早期层中的宝石:通过1000倍输入Token缩减加速长上下文LLM
Discovering the Gems in Early Layers: Accelerating Long-Context LLMs with 1000x Input Token Reduction
AI总结:
提出GemFilter算法,利用LLM早期层作为过滤器选择和压缩输入token,实现2.4倍加速和30%内存减少,显著提升长上下文推理效率且无需训练。
摘要:
大型语言模型(LLMs)在处理长上下文输入方面展现了卓越的能力,但这是以增加计算资源和延迟为代价的。我们的研究引入了一种针对长上下文瓶颈的新方法,以加速LLM推理并减少GPU内存消耗。我们的研究表明,LLM在生成查询答案之前,能够在早期层中识别相关的token。利用这一洞察,我们提出了一种算法,使用LLM的早期层作为过滤器来选择和压缩输入token,显著减少了后续处理的上下文长度。我们的方法GemFilter与现有技术(如标准注意力和SnapKV/H2O)相比,在速度和内存效率上都有显著提升。值得注意的是,与SOTA方法相比,它实现了2.4倍的加速和30%的GPU内存使用减少。在Needle in a Haystack任务上的评估显示,GemFilter显著优于标准注意力和SnapKV,并在LongBench挑战上表现出相当的性能。GemFilter简单、无需训练,且广泛适用于不同的LLM。关键的是,它通过允许人类检查选定的输入序列提供了可解释性。这些发现不仅为LLM部署提供了实际益处,还增强了我们对LLM内部机制的理解,为LLM设计和推理的进一步优化铺平了道路。我们的代码可在\url{https://github.com/SalesforceAIResearch/GemFilter}获取。
英文摘要:
Large Language Models (LLMs) have demonstrated remarkable capabilities in handling long context inputs, but this comes at the cost of increased computational resources and latency. Our research introduces a novel approach for the long context bottleneck to accelerate LLM inference and reduce GPU memory consumption. Our research demonstrates that LLMs can identify relevant tokens in the early layers before generating answers to a query. Leveraging this insight, we propose an algorithm that uses early layers of an LLM as filters to select and compress input tokens, significantly reducing the context length for subsequent processing. Our method, GemFilter, demonstrates substantial improvements in both speed and memory efficiency compared to existing techniques, such as standard attention and SnapKV/H2O. Notably, it achieves a 2.4$\times$ speedup and 30\% reduction in GPU memory usage compared to SOTA methods. Evaluation on the Needle in a Haystack task shows that GemFilter significantly outperforms standard attention, SnapKV and demonstrates comparable performance on the LongBench challenge. GemFilter is simple, training-free, and broadly applicable across different LLMs. Crucially, it provides interpretability by allowing humans to inspect the selected input sequence. These findings not only offer practical benefits for LLM deployment, but also enhance our understanding of LLM internal mechanisms, paving the way for further optimizations in LLM design and inference. Our code is available at \url{https://github.com/SalesforceAIResearch/GemFilter}.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。