Hashrope:一种用于大语言模型上下文管理机制的持久化数据结构
Hashrope: A Persistent Data Structure for Mechanisms of LLM Context Management
AI总结:
Hashrope将大语言模型上下文管理的四种机制统一为单一持久化数据结构,实验显示其在相关工作负载上相比专用结构具有显著性能优势。
摘要:
大语言模型的服务栈通过四种重复出现的机制管理上下文:对不断增长的上下文进行增量编辑、用于树搜索的分支与快照、识别可复用前缀以实现KV缓存复用,以及对重复片段进行编码。目前每种机制都由不同的专用结构提供支持。本文证明,所有这四种机制都可归约为单一的持久化权重平衡rope结构,该结构的每个节点都带有其在梅森素数上的子树的多项式哈希值,因此该结构同时支持对数级编辑和对数级验证内容同一性查询。我们通过预注册的重复实验验证了每种机制,随后与每种机制的已部署专用结构进行了比较:Ropey编辑库、忠实的PagedAttention写时复制块表以及内置的SGLang基数匹配器。在将编辑与同一性查询交错的工作负载上,Hashrope相比Ropey具有无界增长的净优势(在100万字符时为369.7倍)。在使用四个A100 GPU为Qwen2.5-7B-Instruct-1M提供服务的实时vLLM栈上,与基数匹配器相比,Hashrope能准确识别100万token的共享前缀,且速度快1.20倍,而在该上下文长度下,密集引擎甚至无法完成预填充。
英文摘要:
Serving stacks for large language models manage context through four recurring mechanisms: incremental edits to a growing context, branching and snapshotting for tree search, identification of reusable prefixes for KV-cache reuse, and encoding of repeated segments. Each mechanism is served today by a different specialist structure. We show that all four reduce to a single persistent weight-balanced rope whose every node carries a polynomial hash of its subtree over a Mersenne prime, so that the structure supports logarithmic edits and logarithmic verified content-identity queries at the same time. We validate each mechanism with pre-registered, replicated experiments, then compare against a deployed specialist per mechanism: the Ropey editing library, a faithful PagedAttention copy-on-write block table, and the vendored SGLang radix matcher. On workloads that interleave edits with identity queries, Hashrope is net-superior to Ropey by a factor that grows without bound (369.7x at one million characters). Against the radix matcher on a live vLLM stack serving Qwen2.5-7B-Instruct-1M on four A100 GPUs, Hashrope identifies a one-million-token shared prefix exactly and 1.20x faster, at a context length the dense engine cannot even prefill.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。