Graph analytics power a range of applications in areas as diverse as finance, networking and business logistics. A common property of graphs used in the domain of graph analytics is a power-law distribution of vertex connectivity, wherein a small number of vertices are responsible for a high fraction of all connections in the graph. These richly-connected, hot, vertices inherently exhibit high reuse. However, this work finds that state-of-the-art hardware cache management schemes struggle in capitalizing on their reuse due to highly irregular access patterns of graph analytics.
In response, we propose GRASP, domain-specialized cache management at the last-level cache for graph analytics. GRASP augments existing cache policies to maximize reuse of hot vertices by protecting them against cache thrashing, while maintaining sufficient flexibility to capture the reuse of other vertices as needed. GRASP keeps hardware cost negligible by leveraging lightweight software support to pinpoint hot vertices, thus eliding the need for storage-intensive prediction mechanisms employed by state-of-the-art cache management schemes. On a set of diverse graph-analytic applications with large high-skew graph datasets, GRASP outperforms prior domain-agnostic schemes on all datapoints, yielding an average speed-up of 4.2% (max 9.4%) over the best-performing prior scheme. GRASP remains robust on low-/no-skew datasets, whereas prior schemes consistently cause a slowdown.
A Survey on Large Language Model Acceleration based on KV Cache Management
Department of Computing, The Hong Kong Polytechnic University（计算系，香港理工大学）；Department of Computer Science and Engineering（计算机科学与工程系）；Department of Computer Science and Technology（计算机科学与技术系）；Department of Computing and Data Science（计算与数据科学系）
eLLM: Elastic Memory Management Framework for Efficient LLM Serving
Shanghai Jiao Tong University（上海交通大学）；Shanghai Qi Zhi Institute（上海人工智能实验室）；University of Science and Technology of China（中国科学技术大学）；National University of Singapore（新加坡国立大学）
GroupKV: Hierarchical KV Cache Management for Long-Context Diffusion LLM Inference
Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Nanjing University of Science and Technology（南京理工大学）
DCO: Dynamic Cache Orchestration for LLM Accelerators through Predictive Management
Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology（电子与计算机工程系，香港科学与技术大学）；School of Electronic Science and Engineering, Southeast University（电子科学与工程学院，东南大学）
HybriMoE: Hybrid CPU-GPU Scheduling and Cache Management for Efficient MoE Inference
Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）；School of Integrated Circuits, Peking University（北京大学集成电路学院）；Institute of Electronic Design Automation, Peking University（北京大学电子设计自动化研究院）；Beijing Advanced Innovation Center for Integrated Circuits（北京集成电路高精尖创新中心）；School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机科学与工程学院）
