当KV缓存遇上嵌入：用于加速生成式推荐系统服务的动态GPU内存分配
When KV Meets Embeddings: Dynamic GPU Memory Allocation for Accelerating Generative Recommender Serving
AI总结：
RACER通过联合管理GPU内存分配与请求路由，在不牺牲吞吐量的前提下，将生成式推荐系统服务的P99延迟降低24%-38%，SLO满足率达93.5%-99.6%，显著优于现有方案。
摘要：
生成式推荐系统（GR）推理过程中，嵌入热缓存（EMB）与KV缓存会直接竞争有限的GPU高带宽内存（HBM）：为其中一方分配更多内存可提升其效率，但会降低另一方的效率。现有系统对两者进行孤立优化，却忽略了最优EMB-KV分配比例在不同工作负载场景下的波动幅度可达0.35，导致20%-30%的延迟提升空间未被利用。尽管需要在线重新分配来缩小这一差距，但简单的重新分配方法会在关键路径上引入主机到设备（H2D）的重填充流量，引发第99百分位服务水平目标（P99 SLO）违规。为解决该问题，我们提出RACER，它通过两个关键组件在运行时联合管理HBM分配与请求路由：（1）自适应内存分配，一种基于三层近端策略优化（PPO）的控制器（含冻结基础策略、在线残差适配器及突发感知恢复控制器），决策延迟达32微秒，且分配比例与离线最优值的偏差控制在0.024-0.029之间；（2）EMB-KV感知调度，通过联合考虑KV驻留性、嵌入局部性及节点负载来路由请求，避免异构分配下的路由低效。在32节点A100集群上对三个生产级规模数据集的评估显示，与最优静态策略相比，RACER可将P99延迟降低24%-38%，且在稳定、趋势及突发工作负载下实现93.5%-99.6%的SLO满足率，在不牺牲吞吐量的前提下显著优于现有最优基线。
英文摘要：
Generative Recommender (GR) inference places embedding hot caches (EMB) and KV caches in direct competition for limited GPU HBM: allocating more memory to one improves its efficiency but degrades the other. Existing systems optimize them in isolation, overlooking that the optimal EMB-KV allocation ratio can shift by up to 0.35 across workload regimes, leaving 20-30\% latency improvement unrealized. While online reallocation is required to close this gap, naive approaches introduce H2D refill traffic on the critical path, causing P99 SLO violations. To address this, we present RACER, which jointly manages HBM allocation and request routing at runtime through two key components: (1) Adaptive Memory Allocation, a three-layer PPO-based controller (frozen base policy, online residual adapter, and burst-aware recovery controller) that achieves $32\,\mathrm{μs}$ decision latency while staying within 0.024-0.029 of the offline-optimal ratio; and (2) EMB-KV-Aware Scheduling, which routes requests by jointly considering KV residency, embedding locality, and node load to avoid routing inefficiencies under heterogeneous allocations. Evaluations on three production-scale datasets over a 32-node A100 cluster show that RACER reduces P99 latency by 24-38\% over the best static policy and achieves 93.5-99.6\% SLO satisfaction across Steady, Trend, and Burst workloads, significantly outperforming state-of-the-art baselines without sacrificing throughput.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
