LaCache：面向大语言模型高效长上下文建模的阶梯形KV缓存
LaCache: Ladder-Shaped KV Caching for Efficient Long-Context Modeling of Large Language Models
AI总结：
LaCache提出一种无需训练的阶梯形KV缓存与迭代压缩机制，在固定存储预算下同时提升大语言模型的长程建模能力和持续生成效率，实验验证其有效性。
摘要：
大语言模型（LLMs）的最新进展激发了对众多需要强大长程能力的应用的兴趣，这些能力对于处理大量输入上下文和持续生成长输出至关重要。随着序列长度增加，LLMs中键值（KV）对的数量急剧上升，形成了显著的效率瓶颈。本文提出了一种新的KV缓存优化范式，称为LaCache，这是一种无需训练的方法，用于实现LLMs的高效且准确的生成式推理。LaCache使LLMs能够同时应对长程建模中的两个关键挑战：强大的长程能力和在不耗尽内存（OOM）的情况下持续生成。具体而言，LaCache整合了两项关键创新：（1）一种阶梯形KV缓存模式，不仅按顺序（每层内从左到右）存储KV对，而且跨层（从浅层到深层）存储，在固定存储预算下提供扩展的跨度以捕获长程依赖，从而增强长程能力；（2）一种迭代压缩机制，逐步压缩较旧的缓存，在固定缓存大小内为新token释放空间。这种基于token距离的动态压缩能够在受限缓存预算下实现更有效的持续生成。跨各种任务、基准和LLM模型的实验一致验证了LaCache在增强LLMs长程能力方面的有效性。我们的代码可在https://github.com/GATECH-EIC/LaCache获取。
英文摘要：
Recent advancements in Large Language Models (LLMs) have spurred interest in numerous applications requiring robust long-range capabilities, essential for processing extensive input contexts and continuously generating extended outputs. As sequence lengths increase, the number of Key-Value (KV) pairs in LLMs escalates, creating a significant efficiency bottleneck. In this paper, we propose a new KV cache optimization paradigm called LaCache, a training-free method for efficient and accurate generative inference of LLMs. LaCache enables LLMs to simultaneously address both of the critical challenges in long-range modeling: robust long-range capabilities and continuous generation without running out-of-memory (OOM). Specifically, LaCache integrates two key innovations: (1) a ladder-shaped KV cache pattern that stores KV pairs not only sequentially (left-to-right within each layer) but also across layers (from shallow to deep), providing an extended span for capturing long-range dependencies under a fixed storage budget, thereby boosting long-range capabilities; and (2) an iterative compaction mechanism that progressively compresses older caches, freeing up space for new tokens within a fixed cache size. This token distance-based dynamic compression enables more effective continuous generation under constrained cache budgets. Experiments across various tasks, benchmarks, and LLM models consistently validate LaCache's effectiveness in enhancing LLMs' long-range capabilities. Our code is available at https://github.com/GATECH-EIC/LaCache.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
