Large language models (LLMs) are typically served from clusters of GPUs/NPUs that consist of large number of devices. Unfortunately, communication between these devices incurs significant overhead, increasing the inference latency and cost while limiting the scalability. Prior work addressed this issue by overlapping communication with compute, but has severe limitations due to the data dependencies between these operations. In this paper, we propose PRESERVE, a novel framework that prefetches model weights and KV-cache from off-chip HBM memory to the on-chip cache of AI accelerators during the communication operations, which offers various advantages and performance improvements compared to prior methods.
Through extensive experiments conducted on commercial AI accelerators, we demonstrate up to 1.6x end-to-end speedup on state-of-the-art, open-source LLMs. Additionally, we perform a design space exploration that identifies the optimal hardware configuration for the proposed method, showing a further 1.25x improvement in performance per cost by selecting the optimal L2 cache size. Our results show that PRESERVE has the potential to mitigate the memory bottlenecks and communication overheads, offering a solution to improve the performance and scalability of the LLM inference systems.
KV-RM: Regularizing KV-Cache Movement for Static-Graph LLM Serving
Stevens Institute of Technology（史蒂文斯理工学院）；The Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；Argonne National Laboratory（阿贡国家实验室）；Northwestern University（西北大学）
Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization
School of Computing and Information Systems, The University of Melbourne（墨尔本大学计算与信息系统学院）；School of Computer Science and Technology, Huazhong University of Science and Technology（华中科技大学计算机科学与技术学院）
Recency/Frequency Adaptive KV Caching for Large Language Model Serving
Department of Computer Science, Johns Hopkins University, Baltimore, USA（约翰霍普金斯大学计算机科学系）；Argonne National Laboratory, Lemont, USA（阿贡国家实验室）；Parasail, Inc., San Mateo, USA（Parasail公司）
KVServe: Service-Aware KV Cache Compression for Communication-Efficient Disaggregated LLM Serving
University of Chinese Academy of Sciences（中国科学院大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Shanghai Jiao Tong University（上海交通大学）
QCFuse: Query-Aware Cache Fusion via Compressed View for Efficient RAG Serving
Zhejiang University（浙江大学）；East China Normal University（华东师范大学）；Ant Group（蚂蚁集团）；The Hong Kong Polytechnic University（香港理工大学）；Zhejiang Normal University（浙江师范大学）；Tongji University（同济大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；The Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；The Hong Kong University of Science and Technology（香港科学与技术大学）
NVR: Vector Runahead on NPUs for Sparse Memory Access
National Center of Technology Innovation for EDA, School of Integrated Circuits, Southeast University（东南大学集成电路学院、国家EDA技术创新中心）；Houmo AI（后摩智能）；Nanjing University（南京大学）；Huazhong University of Science and Technology（华中科技大学）；Harbin Institute of Technology（哈尔滨工业大学）；Dalian University of Technology（大连理工大学）
ZipMoE: Efficient On-Device MoE Serving via Lossless Compression and Cache-Affinity Scheduling
School of Electronic Science and Engineering, Nanjing University, China.（南京大学电子科学与工程学院）；National Key Laboratory for Novel Software Technology, Nanjing University, China.（南京大学新型软件技术国家重点实验室）
MTServe: Efficient Serving for Generative Recommendation Models with Hierarchical Caches
Institute for Clarity in Documentation（清晰文档研究所）；Inria Paris-Rocquencourt（巴黎-罗quant研究所）；Rajiv Gandhi University（拉吉夫·甘地大学）；Tsinghua University（清华大学）；Palmer Research Laboratories（帕勒实验室）
