arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.04249cs.DCcs.AIcs.LG
收藏

Hermes:面向边缘设备大模型的内存高效流水线推理

Hermes: Memory-Efficient Pipeline Inference for Large Models on Edge Devices

ParisTech Elite Institute of Technology, Shanghai Jiao Tong University(上海交通大学巴黎卓越工程师学院);School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University(上海交通大学电子信息与电气工程学院);UM-SJTU Joint Institute, Shanghai Jiao Tong University(上海交通大学密西根学院);Shanghai Aerospace System Engineering Institute(上海航天系统工程研究所);The University of Melbourne(墨尔本大学)

Xueyuan Han, Zinuo Cai, Yichu Zhang, Chongxin Fan, Junhan Liu, Ruhui Ma, Rajkumar Buyya

AI总结:

针对边缘部署大模型时的内存与延迟问题,提出基于 PIPELOAD 的 Hermes 框架,通过动态内存管理和并行模型加载显著提升推理速度并降低内存占用。

摘要:

近年来,基于 Transformer 的大模型应用取得了众多成功。然而,大模型参数的指数级增长给边缘部署带来了严峻的内存挑战。此前解决该挑战的工作主要集中在优化模型结构和采用内存交换方法上;但前者会降低推理精度,后者则会增加推理延迟。本文介绍了 PIPELOAD,这是一种新颖的内存高效流水线执行机制。它通过引入动态内存管理来降低内存使用,并通过采用并行模型加载来最小化推理延迟。基于 PIPELOAD 机制,我们提出了 Hermes,一个面向边缘设备大模型推理优化的框架。我们在不同规模的基于 Transformer 的模型上评估了 Hermes。实验表明,与最先进的流水线机制相比,Hermes 在 BERT 和 ViT 模型上实现了最高 4.24 倍的推理速度提升和 86.7% 的内存消耗降低;在 GPT 风格模型上实现了 2.58 倍的推理速度提升和 90.3% 的内存消耗降低。

英文摘要:

The application of Transformer-based large models has achieved numerous success in recent years. However, the exponential growth in the parameters of large models introduces formidable memory challenge for edge deployment. Prior works to address this challenge mainly focus on optimizing the model structure and adopting memory swapping methods. However, the former reduces the inference accuracy, and the latter raises the inference latency. This paper introduces PIPELOAD, a novel memory-efficient pipeline execution mechanism. It reduces memory usage by incorporating dynamic memory management and minimizes inference latency by employing parallel model loading. Based on PIPELOAD mechanism, we present Hermes, a framework optimized for large model inference on edge devices. We evaluate Hermes on Transformer-based models of different sizes. Our experiments illustrate that Hermes achieves up to 4.24 X increase in inference speed and 86.7% lower memory consumption than the state-of-the-art pipeline mechanism for BERT and ViT models, 2.58 X increase in inference speed and 90.3% lower memory consumption for GPT-style models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑