DualMap: 为分布式大语言模型服务实现缓存亲和与负载均衡
DualMap: Enabling Both Cache Affinity and Load Balancing for Distributed LLM Serving
AI总结：
DualMap通过双映射策略实现分布式大语言模型服务中的缓存亲和与负载均衡，提升请求容量2.25倍。
摘要：
在大语言模型服务中，重用提示的KV缓存对于减少TTFT和服务成本至关重要。缓存亲和调度，即将具有相同提示前缀的请求定位在一起以最大化KV缓存重用，往往与负载均衡调度冲突，后者将请求均匀分布到计算实例上。现有调度器未能解决这种权衡，因为它们在一个映射空间内操作，通常将缓存亲和路由应用于部分请求，负载均衡路由应用于其余请求，而没有统一的解决方案来实现两个目标。为了解决这一限制，我们提出了DualMap，一种用于分布式大语言模型服务的双映射调度策略，实现了缓存亲和和负载均衡。其核心思想是通过两个独立的哈希函数基于请求提示将每个请求映射到两个候选实例，然后根据当前系统状态智能选择更好的候选。这种设计增加了具有共享前缀的请求被定位在一起的可能性，同时通过"二选一的力量"将不同前缀的请求均匀分布在集群中。为了使DualMap在动态和偏斜的现实工作负载下保持稳健，我们纳入了三种技术：1）SLO-aware请求路由，它优先考虑缓存亲和但当TTFT超过SLO时切换到负载感知调度，从而在不牺牲缓存重用的情况下提高负载平衡；2）热点感知再平衡，它动态地将请求从过载实例迁移到欠载实例，缓解热点并重新平衡系统；3）轻量级双哈希环扩展，它利用双哈希环映射来支持快速且低开销的实例扩展，而无需昂贵的全局重映射。实验表明，与SOTA工作相比，在相同TTFT SLO约束下，DualMap在有效请求容量上提高了多达2.25倍。
英文摘要：
In LLM serving, reusing the KV cache of prompts across requests is critical for reducing TTFT and serving costs. Cache-affinity scheduling, which co-locates requests with the same prompt prefix to maximize KV cache reuse, often conflicts with load-balancing scheduling that distributes requests evenly across compute instances. Existing schedulers fail to reconcile this trade-off as they operate within a single mapping space, typically applying cache-affinity routing to a subset of requests and load-balanced routing to the rest, without a unified solution to achieve both goals. To address this limitation, we propose DualMap, a dual-mapping scheduling strategy for distributed LLM serving that achieves both cache affinity and load balancing. Its key idea is to map each request to two candidate instances via two independent hash functions based on the request prompt, then intelligently select the better candidate based on current system states. This design increases the likelihood that requests with shared prefixes are co-located, while evenly dispersing distinct prefixes across the cluster via ``the power of two choices''. To make DualMap robust under dynamic and skewed real-world workloads, we incorporate three techniques: 1) SLO-aware request routing, which prioritizes cache affinity but switches to load-aware scheduling when TTFT exceeds the SLO, enhancing load balance without sacrificing cache reuse; 2) hotspot-aware rebalancing, which dynamically migrates requests from overloaded to underloaded instances, mitigating hotspots and rebalancing the system; 3) lightweight dual-hash-ring scaling, which leverages a dual-hash-ring mapping to support fast and low-overhead instance scaling without costly global remapping. Experiments on real-world workloads show that DualMap improves effective request capacity by up to 2.25$\times$ under the same TTFT SLO constraints compared with SOTA work.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
