Transformers have emerged as the underpinning architecture for Large Language Models (LLMs). In generative language models, the inference process involves two primary phases: prompt processing and token generation. Token generation, which constitutes the majority of the computational workload, primarily entails vector-matrix multiplications and interactions with the Key-Value (KV) Cache. This phase is constrained by memory bandwidth due to the overhead of transferring weights and KV cache values from the memory system to the computing units. This memory bottleneck becomes particularly pronounced in applications that require long-context and extensive text generation, both of which are increasingly crucial for LLMs.
This paper introduces "Keyformer", an innovative inference-time approach, to mitigate the challenges associated with KV cache size and memory bandwidth utilization. Keyformer leverages the observation that approximately 90% of the attention weight in generative inference focuses on a specific subset of tokens, referred to as "key" tokens. Keyformer retains only the key tokens in the KV cache by identifying these crucial tokens using a novel score function. This approach effectively reduces both the KV cache size and memory bandwidth usage without compromising model accuracy. We evaluate Keyformer's performance across three foundational models: GPT-J, Cerebras-GPT, and MPT, which employ various positional embedding algorithms. Our assessment encompasses a variety of tasks, with a particular emphasis on summarization and conversation tasks involving extended contexts. Keyformer's reduction of KV cache reduces inference latency by 2.1x and improves token generation throughput by 2.4x, while preserving the model's accuracy.
TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selection
University of Science and Technology of China(中国科学技术大学);Tsinghua University(清华大学);Alibaba Cloud Computing(阿里云计算);Xiaohongshu Inc.(小红书公司);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州))
TokenPilot: Cache-Efficient Context Management for LLM Agents
Zhejiang University(浙江大学);University of Electronic Science and Technology of China(电子科技大学);Xi’an University of Electronic Science and Technology(西安电子科技大学);HomologyAI(同源人工智能)
AttentionPredictor: Temporal Patterns Matter for KV Cache Compression
MoE Key Laboratory of Brain-inspired Intelligent Perception and Cognition, University of Science and Technology of China(脑启发智能感知与认知联合实验室,中国科学技术大学);Huawei Noah’s Ark Lab(华为诺亚实验室);College of Intelligence and Computing, Tianjin University(智能与计算学院,天津大学)
Mask Tokens as Prophet: Fine-Grained Cache Eviction for Efficient dLLM Inference
School of Artificial Intelligence, Shanghai Jiao Tong University(上海交通大学人工智能学院);Huazhong University of Science and Technology(华中科技大学);University of Electronic Science and Technology of China(电子科技大学);Carnegie Mellon University(卡内基梅隆大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
Integrative Sciences and Engineering Programme, National University of Singapore(国立新加坡大学整合科学与工程学程);School of Computing, National University of Singapore(国立新加坡大学计算机学院)
Accurate KV Cache Quantization with Outlier Tokens Tracing
School of Computer Science and Technology, Soochow University(苏州大学计算机科学与技术学院);Key Laboratory of Data Intelligence and Advanced Computing, Soochow University(苏州大学数据智能与先进计算重点实验室);Huawei Cloud(华为云)
G-KV: Decoding-Time KV Cache Eviction with Global Attention
School of Computer Science and Technology, Beijing Jiaotong University(计算机科学与技术学院,北京交通大学);MicroSoft(微软);Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence(北京交通数据挖掘与具身智能重点实验室)
Cache-to-Cache: Direct Semantic Communication Between Large Language Models
Tsinghua University(清华大学);Infinigence AI;The Chinese University of Hong Kong(香港中文大学);Shanghai Jiao Tong University(上海交通大学);SLAI;Shanghai AI Laboratory(上海人工智能实验室)
Meta-Soft: Leveraging Composable Meta-Tokens for Context-Preserving KV Cache Compression
Guangdong Institute of Intelligence Science and Technology(广东智能科学与技术研究院);University of Macau(澳门大学);Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Hong Kong University of Science and Technology(香港科技大学)