用 LESS 获得更多：将递归与 KV 缓存压缩相结合以实现高效 LLM 推理
Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference
AI总结：
本文提出 LESS，通过将恒定大小缓存与基于驱逐的 KV 缓存方法结合，在高效推理的同时保留全部 token 信息，显著缩小与全缓存的性能差距。
摘要：
许多计算因素限制了大型语言模型的更广泛部署。在本文中，我们关注由键值（KV）缓存造成的内存瓶颈；KV 缓存是一种计算捷径，需要在解码期间存储先前的 KV 对。现有 KV 缓存方法通过剪枝或驱逐大量相对不那么重要的 KV 对来解决这一问题，以显著降低缓存的内存占用，但在需要回忆大多数先前 token 的任务中，它们的效果可能有限。为缓解该问题，我们提出 LESS，这是一种将（几乎零成本的）恒定大小缓存与基于驱逐的缓存方法进行的简单集成，使所有 token 都能在后续解码步骤中被查询。其随时间保留信息的能力在多种任务上展现出优势：我们证明 LESS 有助于缩小与缓存全部内容之间的性能差距，有时甚至能达到同等性能，同时保持高效。相关代码可在 https://github.com/hdong920/LESS 获取。
英文摘要：
Many computational factors limit broader deployment of large language models. In this paper, we focus on a memory bottleneck imposed by the key-value (KV) cache, a computational shortcut that requires storing previous KV pairs during decoding. While existing KV cache methods approach this problem by pruning or evicting large swaths of relatively less important KV pairs to dramatically reduce the memory footprint of the cache, they can have limited success in tasks that require recollecting a majority of previous tokens. To alleviate this issue, we propose LESS, a simple integration of a (nearly free) constant sized cache with eviction-based cache methods, such that all tokens can be queried at later decoding steps. Its ability to retain information throughout time shows merit on a variety of tasks where we demonstrate LESS can help reduce the performance gap from caching everything, sometimes even matching it, all while being efficient. Relevant code can be found at https://github.com/hdong920/LESS.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
