arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.10087cs.ARcs.DC
收藏

在大语言模型中使用CXL进行恩格拉条件记忆池化

Pooling Engram Conditional Memory in Large Language Models using CXL

Peking University(北京大学);Alibaba Cloud(阿里云);Shandong Yingxin Computer Technology Co., Ltd(山东英信计算机技术有限公司);Renmin University of China(中国人民大学);The University of Hong Kong(香港大学)

Ruiyang Ma, Teng Ma, Zhiyuan Su, Hantian Zha, Xinpeng Zhao, Xuchun Shang, Xingrui Yi, Zheng Liu, Zhu Cao, An Wu, Zhichong Dou, Ziqian Liu, Daikang Kuang, Guojie Luo

AI总结:

本文提出利用CXL内存池存储恩格拉条件记忆,以实现大语言模型的高效存储和低延迟访问。

摘要:

恩格拉条件记忆已成为LLMs中一种有前景的组件,通过将静态知识查找与动态计算解耦。由于恩格拉表现出稀疏的访问模式并支持预取,其庞大的嵌入表非常适合卸载到低层级内存。在本文中,我们提出使用Compute Express Link(CXL)内存池来存储恩格拉。与RDMA相比,CXL提供了细粒度且低延迟的访问,这符合恩格拉最小和离散的检索模式。我们将基于CXL的恩格拉池集成到SGLang中,实现了接近DRAM端到端的性能。这为未来集成恩格拉的LLMs提供了一种可扩展且成本效益高的存储解决方案,而不会影响推理性能。

英文摘要:

Engram conditional memory has emerged as a promising component for LLMs by decoupling static knowledge lookup from dynamic computation. Since Engram exhibits sparse access patterns and supports prefetching, its massive embedding tables are well-suited for offloading to lower-tier memory. In this paper, we propose using Compute Express Link (CXL) memory pool for Engram storage. Compared to RDMA, CXL provides fine-grained and low-latency access required by minimal and discrete retrieval patterns of Engram. We integrate the CXL-based Engram pool into SGLang, achieving near-DRAM end-to-end performance. This provides a scalable and cost-efficient storage solution for future Engram-integrated LLMs without compromising inference performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑