arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.00531cs.DCcs.AI
收藏

TPI-LLM:在低资源边缘设备上高效服务70B规模大语言模型

TPI-LLM: Serving 70B-scale LLMs Efficiently on Low-resource Edge Devices

MBZUAI(穆罕默德·本·扎耶德人工智能大学);UESTC(电子科技大学)

Zonghang Li, Wenjiao Feng, Mohsen Guizani, Hongfang Yu

AI总结:

本文提出TPI-LLM,通过张量并行、滑动窗口内存调度和星型allreduce算法,在低资源边缘设备上高效服务70B大模型,显著降低延迟和内存占用。

摘要:

由于对用户交互数据隐私的担忧,大模型推理正从云端向边缘端转移。然而,边缘设备通常受限于有限的计算能力、内存和带宽,需要多个设备协同运行并加速大语言模型推理。作为主流解决方案的流水线并行在单用户场景下效率低下,而张量并行则面临频繁通信的挑战。本文认为,在低资源设备上,张量并行可以比流水线并行更有效,并提出一个计算和内存高效的张量并行推理系统,名为TPI-LLM,用于服务70B规模模型。TPI-LLM将敏感的原始数据保留在用户设备本地,并引入滑动窗口内存调度器在推理过程中动态管理层权重,使磁盘I/O延迟与计算和通信重叠。这使得更大的模型能够在内存受限的设备上平稳运行。我们分析了通信瓶颈,发现链路延迟而非带宽成为主要问题,因此实现了基于星型的allreduce算法。通过在仿真和真实测试平台上的大量实验,TPI-LLM与Accelerate相比,首token时间和token延迟降低了80%以上,与Transformers和Galaxy相比降低了90%以上,同时将Llama 2-70B的峰值内存占用削减了90%,70B规模模型仅需3.1 GB内存。

英文摘要:

Large model inference is shifting from cloud to edge due to concerns about the privacy of user interaction data. However, edge devices often struggle with limited computing power, memory, and bandwidth, requiring collaboration across multiple devices to run and speed up LLM inference. Pipeline parallelism, the mainstream solution, is inefficient for single-user scenarios, while tensor parallelism struggles with frequent communications. In this paper, we argue that tensor parallelism can be more effective than pipeline on low-resource devices, and present a compute- and memory-efficient tensor parallel inference system, named TPI-LLM, to serve 70B-scale models. TPI-LLM keeps sensitive raw data local in the users' devices and introduces a sliding window memory scheduler to dynamically manage layer weights during inference, with disk I/O latency overlapped with the computation and communication. This allows larger models to run smoothly on memory-limited devices. We analyze the communication bottleneck and find that link latency, not bandwidth, emerges as the main issue, so a star-based allreduce algorithm is implemented. Through extensive experiments on both emulated and real testbeds, TPI-LLM demonstrated over 80% less time-to-first-token and token latency compared to Accelerate, and over 90% compared to Transformers and Galaxy, while cutting the peak memory footprint of Llama 2-70B by 90%, requiring only 3.1 GB of memory for 70B-scale models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑