arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2412.20166cs.ARcs.AI
收藏

PIMphony: 逾越基于PIM的长上下文LLM推理系统的带宽和容量低效问题

PIMphony: Overcoming Bandwidth and Capacity Inefficiency in PIM-based Long-Context LLM Inference System

Hyucksung Kwon, Kyungmo Koo, Janghyeon Kim, Woongkyu Lee, Minjae Lee, Gyeonggeun Jung, Hyungdeok Lee, Yousub Jung, Jaehan Park, Yosub Song, Byeongsu Yang, Haera… 展开作者

Hyucksung Kwon, Kyungmo Koo, Janghyeon Kim, Woongkyu Lee, Minjae Lee, Gyeonggeun Jung, Hyungdeok Lee, Yousub Jung, Jaehan Park, Yosub Song, Byeongsu Yang, Haerang Choi, Guhyun Kim, Jongsoon Won, Woojae Shin, Changhyun Kim, Gyeongcheol Shin, Yongkee Kwon, Ilkon Kim, Euicheol Lim, John Kim, Jungwook Choi

AI总结:

PIMphony通过三种协同设计技术解决基于PIM的长上下文LLM推理系统的带宽和容量低效问题,显著提升吞吐量和效率。

摘要:

长上下文大型语言模型(LLM)的扩展带来了显著的内存系统挑战。虽然处理在内存(PIM)是一种有前途的加速器,但我们发现当扩展到长上下文时,它会面临关键的低效问题:严重的通道利用率不足、限制性能的I/O瓶颈以及大量内存浪费来自静态KV缓存管理。在本工作中,我们提出了PIMphony,一种PIM协调器,通过三种协同设计的技术系统地解决这些问题。首先,基于标记的PIM分区(TCP)确保无论批大小如何都能实现高通道利用率。其次,动态PIM命令调度(DCS)通过重叠数据移动和计算来缓解I/O瓶颈。最后,动态PIM访问(DPA)控制器能够实现动态内存管理,以消除静态内存浪费。通过基于MLIR的编译器实现,并在精确周期模拟器上进行评估,PIMphony显著提高了长上下文LLM推理的吞吐量(最多72B参数和1M上下文长度)。我们的评估显示在仅PIM系统上性能提升最高可达11.3倍,在xPU+PIM系统上性能提升最高可达8.4倍,使LLM在现实世界长上下文应用中的部署更加高效。

英文摘要:

The expansion of long-context Large Language Models (LLMs) creates significant memory system challenges. While Processing-in-Memory (PIM) is a promising accelerator, we identify that it suffers from critical inefficiencies when scaled to long contexts: severe channel underutilization, performance-limiting I/O bottlenecks, and massive memory waste from static KV cache management. In this work, we propose PIMphony, a PIM orchestrator that systematically resolves these issues with three co-designed techniques. First, Token-Centric PIM Partitioning (TCP) ensures high channel utilization regardless of batch size. Second, Dynamic PIM Command Scheduling (DCS) mitigates the I/O bottleneck by overlapping data movement and computation. Finally, a Dynamic PIM Access (DPA) controller enables dynamic memory management to eliminate static memory waste. Implemented via an MLIR-based compiler and evaluated on a cycle-accurate simulator, PIMphony significantly improves throughput for long-context LLM inference (up to 72B parameters and 1M context length). Our evaluations show performance boosts of up to 11.3x on PIM-only systems and 8.4x on xPU+PIM systems, enabling more efficient deployment of LLMs in real-world long-context applications.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑