Large Language Model (LLM) inference uses an autoregressive manner to generate one token at a time, which exhibits notably lower operational intensity compared to earlier Machine Learning (ML) models such as encoder-only transformers and Convolutional Neural Networks. At the same time, LLMs possess large parameter sizes and use key-value caches to store context information. Modern LLMs support context windows with up to 1 million tokens to generate versatile text, audio, and video content. A large key-value cache unique to each prompt requires a large memory capacity, limiting the inference batch size. Both low operational intensity and limited batch size necessitate a high memory bandwidth. However, contemporary hardware systems for ML model deployment, such as GPUs and TPUs, are primarily optimized for compute throughput. This mismatch challenges the efficient deployment of advanced LLMs and makes users pay for expensive compute resources that are poorly utilized for the memory-bound LLM inference tasks.
We propose CENT, a CXL-ENabled GPU-Free sysTem for LLM inference, which harnesses CXL memory expansion capabilities to accommodate substantial LLM sizes, and utilizes near-bank processing units to deliver high memory bandwidth, eliminating the need for expensive GPUs. CENT exploits a scalable CXL network to support peer-to-peer and collective communication primitives across CXL devices. We implement various parallelism strategies to distribute LLMs across these devices. Compared to GPU baselines with maximum supported batch sizes and similar average power, CENT achieves 2.3$\times$ higher throughput and consumes 2.9$\times$ less energy. CENT enhances the Total Cost of Ownership (TCO), generating 5.2$\times$ more tokens per dollar than GPUs.
Pushing the Memory Bandwidth Wall with CXL-enabled Idle I/O Bandwidth Harvesting
School of Electrical;Computer Engineering Georgia Institute of Technology Atlanta, Georgia, USA Email;School of Computer Science Georgia Institute of Technology Atlanta, Georgia, USA Email
Context-Aware Mixture-of-Experts Inference on CXL-Enabled GPU-NDP Systems
Rensselaer Polytechnic Institute(罗切斯特理工学院);University of Massachusetts Amherst(马萨诸塞大学阿默斯特分校);IBM Research Europe(IBM欧洲研究院);IBM T. J. Watson Research Center(IBM托马斯·贾诺斯·沃森研究中心)
TokenStack: A Heterogeneous HBM-PIM Architecture and Runtime for Efficient LLM Inference
Peking University(北京大学);University of Electronic Science and Technology of China(电子科技大学);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所)
PAPI: Exploiting Dynamic Parallelism in Large Language Model Decoding with a Processing-In-Memory-Enabled Computing System
Institute of Computing Technology, CAS(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);King’s College London(伦敦国王学院);ETH Zürich(苏黎世联邦理工学院);University of Toronto(多伦多大学);Vector Institute(向量研究所);NVIDIA(英伟达公司)
PAM: Processing Across Memory Hierarchy for Efficient KV-centric LLM Serving System
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);State Key Lab of Processors(处理器重点实验室)