arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.11521cs.ARcs.DC
收藏

PAM:跨内存层次处理的高效键值中心LLM服务系统

PAM: Processing Across Memory Hierarchy for Efficient KV-centric LLM Serving System

Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);State Key Lab of Processors(处理器重点实验室)

Lian Liu, Shixin Zhao, Yutian Zhou, Yintao He, Mengdi Wang, Yinhe Han, Ying Wang

AI总结:

PAM通过跨内存层次处理,优化键值中心LLM服务系统的带宽与容量,提升效率和可扩展性。

摘要:

大型语言模型(LLM)的广泛应用显著增加了对高效服务系统的需求。随着请求量和上下文长度的增长,键值(KV)相关操作,包括注意力计算和KV缓存存储,已成为关键瓶颈。它们需要大量的内存带宽和容量。不幸的是,现有的LLM服务系统优化于计算密集型工作负载,无法有效处理这些内存密集型操作。即使使用内存计算(PIM)技术,当前单一层次内存设计也无法同时满足带宽和容量需求。为了解决这些挑战,我们提出了Processing Across Memory(PAM),一种以键值为中心的LLM服务系统,协调异构PIM启用的内存设备在一个分层架构中。PAM引入了一种新的计算范式,以平衡高内存带宽和可扩展容量。首先,PAM利用KV访问模式中固有的上下文局部性,智能地将KV令牌分布在内存层次中。其次,为了进一步利用上下文局部性,它引入了PAMattention算法,使细粒度并行注意力计算能够在异构PIM设备上进行。最后,PAM集成了设备内KV映射、设备间KV迁移接口和设备间在线KV调度算法,以动态平衡计算负载。通过同时解决带宽和容量需求,PAM显著提高了LLM服务系统的效率和可扩展性,为大规模AI时代带来了成本效益、高性能的解决方案。

英文摘要:

The widespread adoption of Large Language Models (LLMs) has exponentially increased the demand for efficient serving systems. With growing requests and context lengths, key-value (KV)-related operations, including attention computation and KV cache storage, have emerged as critical bottlenecks. They require massive memory bandwidth and capacity. Unfortunately, existing LLM serving systems, optimized for compute-bound workloads, fail to handle these memory-intensive operations effectively. Even with Processing-In-Memory (PIM) technology, current single-level memory designs cannot simultaneously satisfy the bandwidth and capacity requirements. To address these challenges, we propose Processing Across Memory (PAM), a KV-centric LLM serving system that coordinates heterogeneous PIM-enabled memory devices within a hierarchical architecture. PAM introduces a novel computing paradigm to balance high memory bandwidth with scalable capacity. First, PAM exploits the inherent context locality in KV access patterns to intelligently distribute KV tokens across the memory hierarchy. Second, to further exploit context locality, it introduces the PAMattention algorithm, enabling fine-grained parallel attention computation across heterogeneous PIM devices. Finally, PAM incorporates an intra-device KV mapping, inter-device KV migration interface, and an inter-device online KV scheduling algorithm to dynamically balance computational workloads. By addressing both bandwidth and capacity demands simultaneously, PAM significantly enhances the efficiency and scalability of LLM serving systems, paving the way for cost-effective, high-performance solutions in the era of large-scale AI.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑