arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.07802cs.LGcs.DC
收藏

Kraken:面向高效多设备推理的固有并行Transformer

Kraken: Inherently Parallel Transformers For Efficient Multi-Device Inference

Princeton University(普林斯顿大学)

Rohan Baskar Prabhakar, Hengrui Zhang, David Wentzlaff

AI总结:

针对Transformer自回归推理中并行化带来的集体通信开销高、硬件利用率低的问题,提出Kraken架构,通过层内固定并行度让通信与计算重叠,在保持语言建模性能的同时,将多GPU系统首令牌生成时间平均加快35.6%。

摘要:

大型Transformer网络正越来越多地应用于低推理延迟可提升终端用户体验并催生新应用的场景中。然而,自回归推理资源消耗大,需要并行化才能实现高效运行。并行化会引入集体通信,这类通信不仅开销高昂,还会造成硬件资源利用率低下的阶段。为缓解这一问题,Kraken是标准Transformer架构的演进版本,旨在补充现有张量并行方案,实现多设备系统上的高效推理。通过引入固定程度的层内模型并行度,该架构可让集体操作与计算过程重叠,从而降低延迟并提升硬件利用率。在OpenWebText数据集上训练时,Kraken模型的困惑度与标准Transformer相当,且在SuperGLUE基准测试中保留了原有的语言建模能力。值得注意的是,在使用TensorRT-LLM引擎的多GPU系统上测试时,在一系列模型规模、上下文长度和张量并行度下,Kraken将首令牌生成时间平均加快了35.6%。

英文摘要:

Large Transformer networks are increasingly used in settings where low inference latency can improve the end-user experience and enable new applications. However, autoregressive inference is resource intensive and requires parallelism for efficiency. Parallelism introduces collective communication that is both expensive and represents a phase when hardware resources are underutilized. Towards mitigating this, Kraken is an evolution of the standard Transformer architecture that is designed to complement existing tensor parallelism schemes for efficient inference on multi-device systems. By introducing a fixed degree of intra-layer model parallelism, the architecture allows collective operations to be overlapped with compute, decreasing latency and increasing hardware utilization. When trained on OpenWebText, Kraken models reach a similar perplexity as standard Transformers while also preserving their language modeling capabilities when evaluated on the SuperGLUE benchmark. Importantly, when tested on multi-GPU systems using TensorRT-LLM engines, Kraken speeds up Time To First Token by a mean of 35.6% across a range of model sizes, context lengths, and degrees of tensor parallelism.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑