arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2512.14151cs.ARcs.PF
收藏

基于时间卷积网络的自适应缓存污染控制:用于大语言模型推理工作负载

Adaptive Cache Pollution Control for Large Language Model Inference Workloads Using Temporal CNN-Based Prediction and Priority-Aware Replacement

University of Electronic Science and Technology of China(电子科技大学);Flinders University(弗林德斯大学);Siebel School of Computing and Data Science, University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校西贝尔计算与数据科学学院)

Songze Liu, Hongkun Du, Shaowen Wang

AI总结:

本文提出基于时间卷积网络的自适应缓存污染控制机制,通过预测和优先级替换策略优化LLM推理性能。

摘要:

大型语言模型(LLMs),如GPT和LLaMA,在推理过程中由于频繁的token序列查找和嵌入向量检索,引入了独特的内存访问特征。这些工作负载会产生高度不规则且突发的访问模式,导致传统预取和替换策略预测错误并触发严重的缓存污染,从而降低系统性能。为了解决这一挑战,本文提出了一种自适应缓存污染控制(ACPC)机制,专门针对LLM推理工作负载,集成了基于时间卷积网络(TCN)的访问预测与优先级感知的替换策略。TCN模块学习token访问序列中的时间依赖性以识别潜在高重复的缓存行,而替换策略根据预测的重复可能性和缓存占用情况动态调整驱逐优先级。所提出的框架在具有代表性的transformer-based推理轨迹上实现并评估,包括GPT风格的自回归解码和嵌入检索工作负载。实验结果表明,与最先进的机器学习替换基线相比,ACPC将缓存污染减少了41.7%,提高了缓存命中率8.9%,并实现了L2缺失惩罚的60.0%减少。此外,所提出的基于时间卷积网络的ACPC框架将token生成吞吐量提高了15.9%,并实现了最终损失为0.21的最低值,证实了其在复杂LLM推理工作负载下的优越效率和稳定性。这些结果突显了ACPC在动态LLM访问行为下识别有用缓存行和减少冗余预取的有效性。所提出的方法为优化大规模LLM服务和推理系统中的内存效率和延迟提供了可扩展的、基于学习的解决方案。

英文摘要:

Large Language Models (LLMs), such as GPT and LLaMA, introduce unique memory access characteristics during inference due to frequent token sequence lookups and embedding vector retrievals. These workloads generate highly irregular and bursty access patterns, causing traditional prefetching and replacement policies to mispredict and trigger severe cache pollution, thereby degrading system performance. To address this challenge, this paper proposes an Adaptive Cache Pollution Control (ACPC) mechanism tailored for LLM inference workloads, integrating Temporal Convolutional Network (TCN)-based access prediction with a priority-aware replacement strategy. The TCN module learns temporal dependencies in token access sequences to identify potential high-reuse cache lines, while the replacement policy dynamically adjusts eviction priorities based on predicted reuse likelihood and cache occupancy. The proposed framework is implemented and evaluated on representative transformer-based inference traces, including GPT-style autoregressive decoding and embedding retrieval workloads. Experimental results demonstrate that ACPC reduces cache pollution by 41.7 percent, improves cache hit rate by 8.9 percent, and achieves a 60.0 percent reduction in L2 miss penalty, compared with state-of-the-art machine-learning-based replacement baselines. Additionally, the proposed Temporal CNN-based ACPC framework increases token generation throughput by 15.9 percent and achieves the lowest final loss of 0.21, confirming its superior efficiency and stability under complex LLM inference workloads. These results highlight ACPC's effectiveness in recognizing useful cache lines and mitigating redundant prefetches under dynamic LLM access behaviors. The proposed approach provides a scalable, learning-driven solution for optimizing memory efficiency and latency in large-scale LLM serving and inference systems.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑