Despite the recent success associated with Large Language Models (LLMs), they are notably cost-prohibitive to deploy in resource-constrained environments due to their excessive memory and computational demands. In addition to model parameters, the key-value cache is also stored in GPU memory, growing linearly with batch size and sequence length. As a remedy, recent works have proposed various eviction policies for maintaining the overhead of key-value cache under a given budget. This paper embarks on the efficacy of existing eviction policies in terms of importance score calculation and eviction scope construction. We identify the deficiency of prior policies in these two aspects and introduce RoCo, a robust cache omission policy based on temporal attention scores and robustness measures. Extensive experimentation spanning prefilling and auto-regressive decoding stages validates the superiority of RoCo. Finally, we release EasyKV, a versatile software package dedicated to user-friendly key-value constrained generative inference. Code available at https://github.com/DRSY/EasyKV.
SpindleKV: A Novel KV Cache Reduction Method Balancing Both Shallow and Deep Layers
School of Artificial Intelligence, Wuhan University（武汉大学人工智能学院）；School of Computer Science, Wuhan University（武汉大学计算机学院）；Xiaomi, Beijing, China（小米（北京，中国））；School of Information Management, Wuhan University（武汉大学信息管理学院）
ARM: Attention with Routed-Memory for Learnable Sparse Control
University of Toronto（多伦多大学）；Vector Institute（向量研究所）；Université de Montréal（蒙特利尔大学）；Mila - Quebec AI Institute（Mila-魁北克人工智能研究所）；RIKEN AIP（日本理化学研究所革新智能统合研究中心）；Western University（韦仕敦大学）
