arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2403.01164cs.PFcs.DC
收藏

HeteGen:在资源受限设备上面向大语言模型的异构并行推理

HeteGen: Heterogeneous Parallel Inference for Large Language Models on Resource-Constrained Devices

Xuanlei Zhao, Bin Jia, Haotian Zhou, Ziming Liu, Shenggan Cheng, Yang You

AI总结:

针对资源受限设备上 LLM 推理受 I/O 瓶颈限制的问题,本文提出 HeteGen,利用 CPU/GPU 异构并行与异步重叠降低延迟,推理速度最高较 SOTA 提升逾 317%。

摘要:

近年来,大语言模型(LLM)的出现使模型规模不断增大,给低资源设备上的推理带来挑战。已有方法探索通过卸载来支持低内存推理,但往往因 I/O 瓶颈而效率受损。为在资源受限设备上实现低延迟 LLM 推理,我们提出 HeteGen,这是一种新颖方法,为使用 CPU 和 GPU 的异构并行计算提供了原则性框架。基于该框架,HeteGen 进一步采用异构并行计算和异步重叠来缓解 LLM 的 I/O 瓶颈。实验表明,其推理速度显著提升,最高超过最先进方法 317% 以上。

英文摘要:

In recent times, the emergence of Large Language Models (LLMs) has resulted in increasingly larger model size, posing challenges for inference on low-resource devices. Prior approaches have explored offloading to facilitate low-memory inference but often suffer from efficiency due to I/O bottlenecks. To achieve low-latency LLMs inference on resource-constrained devices, we introduce HeteGen, a novel approach that presents a principled framework for heterogeneous parallel computing using CPUs and GPUs. Based on this framework, HeteGen further employs heterogeneous parallel computing and asynchronous overlap for LLMs to mitigate I/O bottlenecks. Our experiments demonstrate a substantial improvement in inference speed, surpassing state-of-the-art methods by over 317% at most.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑