PackInfer: 用于批量LLM推理的计算和I/O高效注意力
PackInfer: Compute- and I/O-Efficient Attention for Batched LLM Inference
AI总结:
PackInfer通过高效计算和I/O机制优化批量LLM推理,显著降低延迟并提升吞吐量。
摘要:
注意力效率对于大型语言模型(LLM)推理至关重要。尽管先前的进展优化了针对单个请求的注意力执行(例如,FlashAttention),但生产LLM服务依赖于将具有高度异构序列长度的请求进行批处理以提高服务吞吐量。这种不匹配导致严重的计算和I/O不平衡,加剧了慢任务,并且未充分利用GPU资源。我们提出了PackInfer,一个底层的注意力框架,它使异构批量推理的计算和I/O意识执行成为可能。PackInfer将批量请求划分为负载均衡的执行组,通过将多个请求打包到统一的内核启动中,有效饱和GPU利用率。通过在打包的查询-键区域上直接构建注意力内核,PackInfer消除了冗余计算并平衡了线程块执行。然后,它结合了I/O意识的分组,将具有共享前缀的请求定位在一起,并将KV缓存重新组织为组连续的布局,从而减少内存碎片化和冗余数据移动。在真实工作负载上的评估表明,PackInfer将推理延迟减少了13.0-20.1%,并且与最先进的FlashAttention相比,吞吐量提高了20%。
英文摘要:
Attention efficiency is critical to large language model (LLM) inference. While prior advances optimize attention execution for individual requests (e.g., FlashAttention), production LLM serving relies on batching requests with highly heterogeneous sequence lengths for high serving throughput. This mismatch induces severe computation and I/O imbalance, exacerbates stragglers, and underutilizes GPU resources. We present PackInfer, a kernel-level attention framework that enables compute- and I/O-aware execution for heterogeneous batched inference. PackInfer orchestrates batched requests into load-balanced execution groups, effectively saturating GPU utilization by packing multiple requests into unified kernel launches. By constructing attention kernels directly over packed query-key regions, PackInfer eliminates redundant computation and balances thread-block execution. It then incorporates I/O-aware grouping that co-locates shared-prefix requests and reorganizes KV caches into group-contiguous layouts, reducing memory fragmentation and redundant data movement as generation evolves. Evaluations on real-world workloads show that PackInfer reduces inference latency by 13.0-20.1%, and improves throughput by 20% compared to the state-of-the-art FlashAttention.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。