arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2409.11155cs.DCcs.CLcs.LGcs.PF
收藏

ISO:大语言模型推理中序列内计算与通信的重叠

ISO: Overlap of Computation and Communication within Seqenence For LLM Inference

Baichuan Inc.(百川公司)

Bin Xiao, Lei Su

AI总结:

针对大语言模型多GPU张量并行推理中计算与通信串行导致的资源利用率低问题,提出序列级计算-通信重叠策略ISO,在30b/70b模型的预填充阶段实现了显著的效率提升。

摘要:

在大语言模型(Large Language Model, LLM)推理领域,Transformer模型的固有结构结合多GPU张量并行策略,会导致计算与通信按顺序执行。这使得通信阶段计算资源的利用率严重不足。为缓解这一低效问题,已有研究开发了多种技术来优化通信过程中的计算资源利用,这些策略主要包括矩阵计算与通信的重叠,以及跨不同请求的微批次交错执行。尽管如此,这些方法要么无法实现理想的重叠效果,要么在应用上存在一定限制。为克服这些挑战,本文提出了一种在序列层面运行的新型计算-通信重叠策略。该方法不仅提升了重叠程度,还最大限度降低了应用约束。基于30b/70b模型开展的实验评估表明,该方法显著提升了效率。具体而言,在LLM推理的预填充(prefill)阶段,所提出的技术在4090 GPU上可减少约35%的时间消耗,在A800 GPU上可减少约15%的时间消耗。

英文摘要:

In the realm of Large Language Model (LLM) inference, the inherent structure of transformer models coupled with the multi-GPU tensor parallelism strategy leads to a sequential execution of computation and communication. This results in substantial underutilization of computing resources during the communication phase. To mitigate this inefficiency, various techniques have been developed to optimize the use of computational power throughout the communication process. These strategies primarily involve overlapping matrix computations and communications, as well as interleaving micro-batches across different requests. Nonetheless, these approaches either fall short of achieving ideal overlap or impose certain limitations on their application. To overcome these challenges, this paper introduces a novel strategy for computation-communication overlap that operates at the sequence level. This method not only enhances the degree of overlap but also minimizes the constraints on its applicability. Experimental evaluations conducted using 30b/70b models have demonstrated significant improvements in efficiency. Specifically, the proposed technique has been shown to reduce time consumption by approximately 35% on 4090 GPU and by roughly 15% on A800 GPU during the prefill stage of LLM inference.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑