SnapStream: 在数据流加速器上高效处理长序列解码
SnapStream: Efficient Long Sequence Decoding on Dataflow Accelerators
AI总结:
本文探讨了在数据流加速器上实现长序列解码的效率与准确性,提出SnapStream方法,通过压缩KV缓存提升芯片内存利用率,减少精度损失。
摘要:
1000亿参数以上的大型语言模型(LLMs)和10万+上下文长度的支持,导致对芯片内内存的需求增加。StreamingLLM和SnapKV展示了如何在保持模型精度的同时控制KV缓存大小。然而,这些技术在工业部署中使用vLLM或SGLang等框架时并不常见。原因有两个方面:一方面,这些框架采用静态图和连续分批方法,使修改标准多头注意力算法变得困难;另一方面,这些技术对现代指令跟随和推理模型的准确性影响尚不明确,模糊了实施这些技术的必要性。本文探讨了这些准确性影响,并开发了SnapStream,一种可扩展部署的KV缓存压缩方法。在SambaNova SN40L加速器上16路张量并行部署DeepSeek-671B时,展示了SnapStream在128k上下文长度和每秒1832个token的现实生产环境中的有效性。SnapStream实现了4倍的芯片内存利用率,并在LongBench-v2、AIME24和LiveCodeBench上引入了最小的精度损失。据我们所知,这是首次在具有静态图和连续分批的生产推理系统中实现稀疏KV注意力技术。
英文摘要:
The proliferation of 100B+ parameter Large Language Models (LLMs) with 100k+ context length support have resulted in increasing demands for on-chip memory to support large KV caches. Techniques such as StreamingLLM and SnapKV demonstrate how to control KV cache size while maintaining model accuracy. Yet, these techniques are not commonly used within industrial deployments using frameworks like vLLM or SGLang. The reason is twofold: on one hand, the static graphs and continuous batching methodology employed by these frameworks make it difficult to admit modifications to the standard multi-head attention algorithm, while on the other hand, the accuracy implications of such techniques on modern instruction-following and reasoning models are not well understood, obfuscating the need for implementing these techniques. In this paper, we explore these accuracy implications on Llama-3.1-8B-Instruct and DeepSeek-R1, and develop SnapStream, a KV cache compression method that can be deployed at scale. We demonstrate the efficacy of SnapStream in a 16-way tensor-parallel deployment of DeepSeek-671B on SambaNova SN40L accelerators running at 128k context length and up to 1832 tokens per second in a real production setting. SnapStream enables $4\times$ improved on-chip memory usage and introduces minimal accuracy degradation on LongBench-v2, AIME24 and LiveCodeBench. To the best of our knowledge, this is the first implementation of sparse KV attention techniques deployed in a production inference system with static graphs and continuous batching.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。