arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2511.02168cs.DCcs.LG
收藏

消除多GPU性能税:高效分布式LLM的系统方法

Eliminating Multi-GPU Performance Taxes: A Systems Approach to Efficient Distributed LLMs

Octavian Alexandru Trifan, Karthik Sangaiah, Muhammad Awad, Muhammad Osama, Sumanth Gudaparthi, Alexandru Nicolau, Alexander Veidenbaum, Ganesh Dasika

AI总结:

本文提出超越传统BSP模型,利用Iris for Triton库的内核内通信原语设计细粒度编程模式,消除多GPU分布式LLM执行中的三大性能税,实现端到端延迟10-20%的加速。

摘要:

随着大型语言模型(LLMs)持续扩展,其工作负载越来越依赖于跨多个GPU的分布式执行。然而,此类场景中使用的传统整体同步并行(BSP)模型引入了显著的性能低效问题。为了刻画这些瓶颈,我们引入了“三大税”(整体同步、内核间数据局部性和内核启动开销)作为分析框架。我们提出超越僵化的BSP模型,以解决分布式GPU执行中的关键低效问题。通过利用Iris for Triton等库,我们获得了内核内通信原语,从而能够设计出新颖的细粒度编程模式,提供比传统基于BSP的方法更大的灵活性和更高的性能。这些模式通过创建直接的、tile级生产者-消费者流水线,并用细粒度数据流同步替代全局屏障,系统地消除了三大税。将该方法应用于关键内核,从基础的All-Gather + 通用矩阵乘法操作到复杂的Flash Decode算法,我们观察到端到端延迟相比基于BSP的方法加速了10-20%,为分布式LLM工作负载确立了更可编程且更高效的范式。

英文摘要:

As large language models (LLMs) continue to scale, their workloads increasingly rely on distributed execution across multiple GPUs. However, the conventional bulk synchronous parallel~(BSP) model used in such settings introduces significant performance inefficiencies. To characterize these bottlenecks, we introduce the ''Three Taxes'' (Bulk Synchronous, Inter-Kernel Data Locality, and Kernel Launch Overhead) as an analytical framework. We propose moving beyond the rigid BSP model to address key inefficiencies in distributed GPU execution. By exploiting libraries like Iris for Triton, we gain access to in-kernel communication primitives that enable the design of novel fine-grained programming patterns, offering greater flexibility and performance than traditional BSP-based approaches. These patterns systematically eliminate the three taxes by creating direct, tile-level producer-consumer pipelines and replacing global barriers with fine-grained dataflow synchronization. Applying this methodology to critical kernels, from the foundational All-Gather + general matrix multiplication operation to the complex Flash Decode algorithm, we observe a 10-20% speedup in end-to-end latency over BSP-based approaches, establishing a more programmable and efficient paradigm for distributed LLM workloads.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑