arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.20041cs.CL
收藏

Transformer-Lite:大语言模型在手机GPU上的高效部署

Transformer-Lite: High-efficiency Deployment of Large Language Models on Mobile Phone GPUs

OPPO AI Center(OPPO AI中心)

Luchang Li, Sheng Qian, Jie Lu, Lunxi Yuan, Rui Wang, Qin Xie

AI总结:

本文提出四项优化技术并实现于移动推理引擎Transformer-Lite,在手机GPU上大幅提升大语言模型推理速度,预填充加速超10倍、解码加速2到3倍。

摘要:

大语言模型(LLM)在手机上被广泛用于智能助手、文本摘要、翻译和多模态等任务。然而,当前端侧LLM部署方法的推理速度较慢,导致用户体验不佳。为促进LLM在设备GPU上的高效部署,我们提出四项优化技术:(a)一种基于符号表达式的方法,用于支持动态形状模型推理;(b)算子优化与执行优先级设置,以提升推理速度并减少手机卡顿;(c)一种名为M0E4的FP4量化方法,用于降低反量化开销;(d)一种基于子张量的技术,用于消除LLM推理后复制KV缓存的必要性。此外,我们在移动推理引擎Transformer-Lite中实现了这些方法,该引擎兼容高通和联发科处理器。我们使用架构和参数量各异(从2B到14B)的LLM评估了Transformer-Lite的性能。具体而言,对于ChatGLM2 6B,我们实现了121 token/s的预填充速度和14 token/s的解码速度;对于较小的Gemma 2B,分别实现了330 token/s和30 token/s的速度。与基于CPU的FastLLM和基于GPU的MLC-LLM相比,我们的引擎在预填充速度上实现了超过10倍的加速,在解码速度上实现了2到3倍的加速。

英文摘要:

The Large Language Model (LLM) is widely employed for tasks such as intelligent assistants, text summarization, translation, and multi-modality on mobile phones. However, the current methods for on-device LLM deployment maintain slow inference speed, which causes poor user experience. To facilitate high-efficiency LLM deployment on device GPUs, we propose four optimization techniques: (a) a symbolic expression-based approach to support dynamic shape model inference; (b) operator optimizations and execution priority setting to enhance inference speed and reduce phone lagging; (c) an FP4 quantization method termed M0E4 to reduce dequantization overhead; (d) a sub-tensor-based technique to eliminate the need for copying KV cache after LLM inference. Furthermore, we implement these methods in our mobile inference engine, Transformer-Lite, which is compatible with both Qualcomm and MTK processors. We evaluated Transformer-Lite's performance using LLMs with varied architectures and parameters ranging from 2B to 14B. Specifically, we achieved prefill and decoding speeds of 121 token/s and 14 token/s for ChatGLM2 6B, and 330 token/s and 30 token/s for smaller Gemma 2B, respectively. Compared with CPU-based FastLLM and GPU-based MLC-LLM, our engine attains over 10x speedup for the prefill speed and 2~3x speedup for the decoding speed.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑