arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.02737cs.AR
收藏

Ouroboros: 以片上SRAM为基础的片上计算内存架构,采用令牌粒度流水线实现大语言模型推理

Ouroboros: Wafer-Scale SRAM CIM with Token-Grained Pipelining for Large Language Model Inference

SKLP, Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学)

Yiqi Liu, Yudong Pan, Mengdi Wang, Shixin Zhao, Haonan Zhu, Yinhe Han, Lei Zhang, Ying Wang

AI总结:

Ouroboros通过令牌粒度流水线、分布式动态KV缓存管理和通信感知映射,实现了大语言模型推理在片上SRAM中的高效执行,提升了吞吐量和能效。

摘要:

传统LLM推理架构由于频繁的内存层次间数据移动导致能耗和延迟较高。我们提出了Ouroboros,一种基于片上SRAM的计算内存(CIM)架构,所有操作均在本地执行,消除了芯片外迁移。为了最大化其有限的一级容量,我们引入了三个创新:令牌粒度流水线:取代序列级流水线以缓解长度变化,提高利用率并减少激活存储。分布式动态KV缓存管理:将内存与计算分离,利用碎片化的SRAM实现高效的KV存储。通信感知映射:优化核心分配以在晶圆上实现局部性和容错性。实验结果表明,Ouroboros在吞吐量和能耗效率上分别实现了平均4.1倍和4.2倍的提升,对于13B模型峰值分别达到9.1倍和17倍。(*由于arXiv通知“摘要字段不能超过1920个字符”,出现的摘要被缩短。如需完整摘要,请下载文章。)

英文摘要:

Conventional LLM inference architectures suffer from high energy and latency due to frequent data movement across memory hierarchies. We propose Ouroboros, a wafer-scale SRAM-based Computing-in-Memory (CIM) architecture that executes all operations in situ, eliminating off-chip migration. To maximize its limited first-level capacity, we introduce three innovations: Token-Grained Pipelining: Replaces sequence-level pipelining to mitigate length variations, boosting utilization and reducing activation storage. Distributed Dynamic KV Cache Management: Decouples memory from compute to leverage fragmented SRAM for efficient KV storage. Communication-Aware Mapping: Optimizes core allocation for locality and fault tolerance across the wafer. Experimental results show Ouroboros achieves average gains of $4.1\times$ in throughput and $4.2\times$ in energy efficiency, peaking at $9.1\times$ and $17\times$ for the 13B model. (*Due to the notification of arXiv "The Abstract field cannot be longer than 1,920 characters", the appeared Abstract is shortened. For the full Abstract, please download the Article.)

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑