KVDirect：分布式解耦式LLM推理
KVDirect: Distributed Disaggregated LLM Inference
AI总结：
KVDirect通过优化KV缓存传输，实现分布式解耦式LLM推理，采用张量通信、自定义通信库和拉取式传输策略，将每请求延迟降低55%。
摘要：
大型语言模型（LLM）已成为许多应用的新基础，像风暴一样重塑人类社会。解耦式推理将预填充和解码阶段分离，是提高硬件利用率和服务质量的一种有前景的方法。然而，由于节点间通信效率低下，现有系统将解耦式推理限制在单节点内，限制了资源分配的灵活性并降低了服务容量。本文介绍了KVDirect，它优化了KV缓存传输以实现分布式解耦式LLM推理。KVDirect通过以下贡献实现这一目标。首先，我们提出了一种新颖的基于张量的通信机制，减少了传统分布式GPU系统中的同步开销。其次，我们设计了一个自定义通信库，以支持动态GPU资源调度和高效的KV缓存传输。第三，我们引入了一种基于拉取的KV缓存传输策略，减少了GPU资源空闲并改善了延迟。最后，我们将KVDirect实现为一个开源LLM推理框架。我们的评估表明，在相同资源约束下，KVDirect在各种工作负载中相比基线将每请求延迟降低了55%。
英文摘要：
Large Language Models (LLMs) have become the new foundation for many applications, reshaping human society like a storm. Disaggregated inference, which separates prefill and decode stages, is a promising approach to improving hardware utilization and service quality. However, due to inefficient inter-node communication, existing systems restrict disaggregated inference to a single node, limiting resource allocation flexibility and reducing service capacity. This paper introduces KVDirect, which optimizes KV cache transfer to enable a distributed disaggregated LLM inference. KVDirect achieves this through the following contributions. First, we propose a novel tensor-centric communication mechanism that reduces the synchronization overhead in traditional distributed GPU systems. Second, we design a custom communication library to support dynamic GPU resource scheduling and efficient KV cache transfer. Third, we introduce a pull-based KV cache transfer strategy that reduces GPU resource idling and improves latency. Finally, we implement KVDirect as an open-source LLM inference framework. Our evaluation demonstrates that KVDirect reduces per-request latency by 55% compared to the baseline across diverse workloads under the same resource constraints.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
