arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.38809cs.LG
收藏

CPU上的因果注意力:一种用于预填充的缓存感知且负载均衡的调度内核

Causal Attention on CPU: A Cache-Aware and Load-Balanced Scheduling Kernel for Prefill

Intel Corp.(英特尔公司)

Kwun Fung Lau, Frost Mitchell, Kanya Mo, Emani Ashok

AI总结:

针对多智能体并发长上下文预填充耗尽GPU资源的问题,提出CACPU缓存感知负载均衡调度内核,集成到SGLang后可加速Llama-3.1-8B的首token生成时间并保持精度。

摘要:

智能体AI在多个领域带来显著生产力提升,但多个智能体的并发长上下文预填充会耗尽GPU计算资源。尽管CPU是可行替代方案,现有CPU注意力内核无法适配多层缓存层级且缺乏有效线程调度。为此,我们提出CACPU,一种高性能注意力内核,具有分层缓存分块和专为因果全序列预填充设计的内核内任务调度机制。CACPU相比PyTorch SDPA、FlexAttention和SGLang注意力内核实现最高2.01倍加速。集成到SGLang后,其将Llama-3.1-8B的首token生成时间(TTFT)降低最高1.74倍,同时保持模型精度。

英文摘要:

Agentic AI drives significant productivity gains across diverse domains, but concurrent long-context prefills from multiple agents can saturate GPU compute resources. Although CPUs offer a viable alternative, existing CPU attention kernels fall short in adapting to multi-tier cache hierarchies and lack effective thread scheduling. To address this, we introduce CACPU, a high-performance attention kernel featuring tiered cache blocking and an in-kernel task dispatching mechanism designed for causal full-sequence prefilling. CACPU achieves up to a 2.01x speedup over PyTorch SDPA, FlexAttention, and SGLang attention kernels. Integrated into SGLang, it reduces Time to First Token (TTFT) of Llama-3.1-8B by up to 1.74x while preserving model accuracy.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑