Harvest：机会性点对点GPU缓存用于LLM推理
Harvest: Opportunistic Peer-to-Peer GPU Caching for LLM Inference
AI总结：
Harvest通过利用点对点GPU互连技术，动态管理GPU内存，减少数据移动开销，显著提升LLM推理的吞吐量。
摘要：
大型语言模型（LLM）推理日益受到GPU内存容量的限制，而非计算吞吐量，这主要是由于模型规模的增长以及自回归解码过程中键值（KV）缓存的线性增长所驱动的。现有方法通过将模型状态和KV张量卸载到主机内存来缓解内存压力，但受限于PCIe带宽，导致了显著的延迟。我们提出了Harvest，一种机会性GPU缓存管理框架，利用高带宽点对点GPU互连技术，动态地将模型权重和KV缓存放置在未使用的GPU内存中。Harvest将同机GPU内存视为一个临时缓存层级，在动态内存可用性下保持正确性的同时减少数据移动开销。我们通过使用Harvest加速两个广泛使用的推理组件——专家层权重和KV缓存条目，实现了超过2倍的吞吐量提升。
英文摘要：
Large Language Model (LLM) inference is increasingly constrained by GPU memory capacity rather than compute throughput, driven by growing model sizes and the linear growth of the key-value (KV) cache during autoregressive decoding. Existing approaches mitigate memory pressure by offloading model state and KV tensors to host memory, but incur substantial latency due to limited PCIe bandwidth. We present Harvest, an opportunistic GPU cache management framework that exploits high-bandwidth peer-to-peer GPU interconnects to dynamically place model weights and KV cache in unused GPU memory. Harvest treats peer GPU memory as a transient cache tier, preserving correctness while reducing data movement overhead under dynamic memory availability. We demonstrate significant throughput speedup of more than 2 times by using Harvest to accelerate the retrieval of two widely-used inference components: expert layer weights and KV cache entries.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
