在大语言模型中使用CXL进行恩格拉条件记忆池化
Pooling Engram Conditional Memory in Large Language Models using CXL
AI总结:
本文提出利用CXL内存池存储恩格拉条件记忆,以实现大语言模型的高效存储和低延迟访问。
摘要:
恩格拉条件记忆已成为LLMs中一种有前景的组件,通过将静态知识查找与动态计算解耦。由于恩格拉表现出稀疏的访问模式并支持预取,其庞大的嵌入表非常适合卸载到低层级内存。在本文中,我们提出使用Compute Express Link(CXL)内存池来存储恩格拉。与RDMA相比,CXL提供了细粒度且低延迟的访问,这符合恩格拉最小和离散的检索模式。我们将基于CXL的恩格拉池集成到SGLang中,实现了接近DRAM端到端的性能。这为未来集成恩格拉的LLMs提供了一种可扩展且成本效益高的存储解决方案,而不会影响推理性能。
英文摘要:
Engram conditional memory has emerged as a promising component for LLMs by decoupling static knowledge lookup from dynamic computation. Since Engram exhibits sparse access patterns and supports prefetching, its massive embedding tables are well-suited for offloading to lower-tier memory. In this paper, we propose using Compute Express Link (CXL) memory pool for Engram storage. Compared to RDMA, CXL provides fine-grained and low-latency access required by minimal and discrete retrieval patterns of Engram. We integrate the CXL-based Engram pool into SGLang, achieving near-DRAM end-to-end performance. This provides a scalable and cost-efficient storage solution for future Engram-integrated LLMs without compromising inference performance.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。