arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2604.15464cs.PFcs.AIcs.LG
收藏

不规则分页注意机制:一种高性能且灵活的TPU大语言模型推理内核

Ragged Paged Attention: A High-Performance and Flexible LLM Inference Kernel for TPU

Google(谷歌)

Jevin Jiang, Ying Chen, Blake A. Hechtman, Fenghui Zhang, Yarong Mu

AI总结:

本文提出RPA,一种用于TPU的高性能注意力内核,通过细粒度分块、定制软件流水线和分布感知编译策略,提升TPU在动态不规则执行模式下的效率,实现在解码和预填充任务中达到86%和73%的利用率。

摘要:

大语言模型(LLM)部署正逐渐转向如谷歌的张量处理单元(TPU)等高效能加速器,注重性能和总拥有成本(TCO)。然而,现有LLM推理内核和服务系统仍以GPU为中心,缺乏有效映射LLM工作负载到TPU架构的方法,特别是在现代服务中常见的动态和不规则执行模式。本文提出Ragged Paged Attention(RPA),一种用于TPU的高性能且灵活的注意力内核,使用Pallas和Mosaic实现。RPA通过三个关键技术解决这些挑战:(1)细粒度分块以实现对不规则内存的高效动态切片;(2)一个定制的软件流水线,将键值缓存更新与注意力计算融合;(3)一种分布感知的编译策略,为解码、预填充和混合工作负载生成专用内核。在TPU7x上评估Llama 3 8B时,RPA在解码任务中达到高达86%的内存带宽利用率(MBU),在预填充任务中达到73%的模型FLOPs利用率(MFU)。集成到vLLM和SGLang的主要TPU后端中,RPA为高效TPU推理提供了生产级基础,并提供了关于内核设计的实用见解。

英文摘要:

Large Language Model (LLM) deployment is increasingly shifting to cost-efficient accelerators like Google's Tensor Processing Units (TPUs), prioritizing both performance and total cost of ownership (TCO). However, existing LLM inference kernels and serving systems remain largely GPU-centric, and there is no well-established approach for efficiently mapping LLM workloads onto TPU architectures--particularly under the dynamic and ragged execution patterns common in modern serving. In this paper, we present Ragged Paged Attention (RPA), a high-performance and flexible attention kernel for TPUs, implemented using Pallas and Mosaic. RPA addresses these challenges through three key techniques: (1) fine-grained tiling to enable efficient dynamic slicing over ragged memory, (2) a custom software pipeline that fuses KV cache updates with attention computation, and (3) a distribution-aware compilation strategy that generates specialized kernels for decode, prefill, and mixed workloads. Evaluated on Llama 3 8B on TPU7x, RPA achieves up to 86% memory bandwidth utilization (MBU) in decode and 73% model FLOPs utilization (MFU) in prefill. Integrated as the primary TPU backend in vLLM and SGLang, RPA provides a production-grade foundation for efficient TPU inference and offers practical insights into kernel design.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑