Agentic AI drives significant productivity gains across diverse domains, but concurrent long-context prefills from multiple agents can saturate GPU compute resources. Although CPUs offer a viable alternative, existing CPU attention kernels fall short in adapting to multi-tier cache hierarchies and lack effective thread scheduling. To address this, we introduce CACPU, a high-performance attention kernel featuring tiered cache blocking and an in-kernel task dispatching mechanism designed for causal full-sequence prefilling. CACPU achieves up to a 2.01x speedup over PyTorch SDPA, FlexAttention, and SGLang attention kernels. Integrated into SGLang, it reduces Time to First Token (TTFT) of Llama-3.1-8B by up to 1.74x while preserving model accuracy.
TideKV: Delayed KV Eviction via Layer-Wise Chunked Prefill for Memory-Efficient Long-Context LLM Inference
Northwestern Polytechnical University(西北工业大学);Xi’an Aeronautics Computing Technique Research Institute, AVIC(中航工业西安航空计算技术研究所);College of Computer Science and Technology, Harbin Engineering University(哈尔滨工程大学计算机科学与技术学院)
RelayCaching: Accelerating LLM Collaboration via Decoding KV Cache Reuse
Beijing University of Posts(北京邮电大学);Tsinghua University, Beijing, China(清华大学);University of Electronic Science(电子科学大学);Peking University, Beijing, China(北京大学)
Achieving Cloud-Grade SLOs for Local Mixture-of-Experts Inference through CPU-GPU Hybrid Design
Tsinghua University(清华大学);Xingyun Integrated Circuits Co., Ltd.(星云集成电路有限公司);Beijing National Research Center for Information Science and Technology(北京信息科学与技术国家研究中心)
TPLA: Tensor Parallel Latent Attention for Efficient Disaggregated Prefill and Decode Inference
Institute for Artificial Intelligence, Peking University(北京大学人工智能研究所);State Key Laboratory of General Artificial Intelligence, BIGAI(通用人工智能国家重点实验室,BIGAI);Tencent Youtu Lab, Shanghai, China(腾讯云智实验室,上海,中国)
Integrative Sciences and Engineering Programme, National University of Singapore(国立新加坡大学整合科学与工程学程);School of Computing, National University of Singapore(国立新加坡大学计算机学院)
HybriMoE: Hybrid CPU-GPU Scheduling and Cache Management for Efficient MoE Inference
Institute for Artificial Intelligence, Peking University(北京大学人工智能研究院);School of Integrated Circuits, Peking University(北京大学集成电路学院);Institute of Electronic Design Automation, Peking University(北京大学电子设计自动化研究院);Beijing Advanced Innovation Center for Integrated Circuits(北京集成电路高精尖创新中心);School of Computer Science and Engineering, Beihang University(北京航空航天大学计算机科学与工程学院)