FlashInfer：面向LLM推理服务的高效可定制注意力引擎
FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving
AI总结：
FlashInfer通过块稀疏格式、可定制模板和负载均衡调度，为LLM推理提供高效注意力引擎，显著降低延迟并提升性能。
摘要：
Transformer模型由注意力机制驱动，构成了大型语言模型（LLM）的基础。随着这些模型规模的扩大，高效的GPU注意力内核对于实现高吞吐量和低延迟推理变得至关重要。多样化的LLM应用需要灵活且高性能的注意力解决方案。我们提出了FlashInfer：一个用于LLM服务的可定制且高效的注意力引擎。FlashInfer利用块稀疏格式和可组合格式来处理KV缓存存储异构性，以优化内存访问并减少冗余。它还提供了可定制的注意力模板，通过即时（JIT）编译实现对各种设置的适配。此外，FlashInfer的负载均衡调度算法能够适应用户请求的动态变化，同时保持与需要静态配置的CUDAGraph的兼容性。FlashInfer已被集成到SGLang、vLLM和MLC-Engine等领先的LLM服务框架中。全面的内核级和端到端评估表明，FlashInfer能够在各种推理场景中显著提升内核性能：与最先进的LLM服务解决方案相比，FlashInfer在LLM服务基准测试中实现了29-69%的令牌间延迟降低，在长上下文推理中实现了28-30%的延迟降低，并在并行生成的LLM服务中实现了13-17%的加速。
英文摘要：
Transformers, driven by attention mechanisms, form the foundation of large language models (LLMs). As these models scale up, efficient GPU attention kernels become essential for high-throughput and low-latency inference. Diverse LLM applications demand flexible and high-performance attention solutions. We present FlashInfer: a customizable and efficient attention engine for LLM serving. FlashInfer tackles KV-cache storage heterogeneity using block-sparse format and composable formats to optimize memory access and reduce redundancy. It also offers a customizable attention template, enabling adaptation to various settings through Just-In-Time (JIT) compilation. Additionally, FlashInfer's load-balanced scheduling algorithm adjusts to dynamism of user requests while maintaining compatibility with CUDAGraph which requires static configuration. FlashInfer have been integrated into leading LLM serving frameworks like SGLang, vLLM and MLC-Engine. Comprehensive kernel-level and end-to-end evaluations demonstrate FlashInfer's ability to significantly boost kernel performance across diverse inference scenarios: compared to state-of-the-art LLM serving solutions, FlashInfer achieve 29-69% inter-token-latency reduction compared to compiler backends for LLM serving benchmark, 28-30% latency reduction for long-context inference, and 13-17% speedup for LLM serving with parallel generation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
