AsyncSpade:基于异步稀疏解码的高效测试时扩展
AsyncSpade: Efficient Test-Time Scaling with Asynchronous Sparse Decoding
AI总结:
AsyncSpade通过异步解耦KV缓存过滤与自回归解码,利用轻量时间回归预测查询状态,在长CoT测试时扩展中消除顺序依赖,显著降低TPOT并保持模型精度。
摘要:
测试时扩展(TTS)通过长思维链(CoT)提升大语言模型的推理能力,但线性增长的KV缓存加剧了LLM解码的内存瓶颈。查询感知的页级稀疏解码在受限FLOPs预算下可实现最先进的性能,但受限于顺序依赖的页过滤和粗粒度的令牌选择,在高并发和长CoT场景下(其运行时甚至高于前向管线本身)损害了服务效率和TTS任务的模型性能。本文首先发现,当前步的查询状态可以从近期查询的短窗口中统一且精确地近似,从而无需在解码循环中等待即可实现免训练的查询感知稀疏性。我们提出AsyncSpade,一个用于高效TTS的异步框架,包含两个核心组件:(1)一个新颖的轻量级时间回归模块,用于预测下一令牌的查询状态;(2)一个异步且解耦的框架,将KV缓存过滤从自回归解码循环中解耦,通过异步性将令牌级KV选择与前向推理计算重叠。据我们所知,AsyncSpade是首个在不牺牲模型性能的情况下消除顺序依赖的方法。我们在配备A100节点的常见LLM服务设置上验证了AsyncSpade的有效性,其中AsyncSpade将KV缓存操作与推理管线完全重叠,实现了理论最优的每令牌输出时间(TPOT)。具体而言,与最先进基线(即Quest)相比,AsyncSpade在Qwen3-8B和Qwen3-32B模型上将TPOT降低了超过20%,与全注意力相比至少降低50%,同时在各种TTS基准(AIME-24/25、GPQA-Diamond、MATH-500)上达到或超越其准确率。
英文摘要:
Test-time scaling (TTS) boosts LLM reasoning via long chain-of-thought (CoT), but the linear KV-cache growth amplifies the memory-bound bottleneck of LLM decoding. Query-aware page-level sparse decoding can achieve state-of-the-art performance under constrained FLOPs budgets, but is limited by both sequential-dependent page filtering and coarse-grained token selection, hampering serving efficiency and model performance on TTS tasks under high concurrency and long CoT scenarios (consuming even higher runtime than the forward pipeline itself). In this paper, we first find that the current-step query state can be accurately approximated in a unified manner from a short window of recent queries, enabling training-free query-aware sparsity without waiting in the decoding loop. We propose AsyncSpade, an asynchronous framework for efficient TTS built on two core components: (1) a novel light-weight temporal-regressive module that predicts the next-token query state; (2) an asynchronous and disaggregated framework that decouples the KV cache filtering from the auto-regressive decoding loop, overlapping the token-level KV selection with the forward inference computation through asynchronism. To our knowledge, AsyncSpade is the first to eliminate the sequential dependence without sacrificing model performance. We validate the effectiveness of AsyncSpade on common LLM serving setups with an A100 node, where AsyncSpade fully overlaps KV-cache operations with the inference pipeline, achieving theoretical optimal time-per-output-token (TPOT). Specifically, AsyncSpade delivers over 20% reduction on TPOT compared to SoTA baseline (i.e. Quest) and at least 50% TPOT reduction compared to full attention on Qwen3-8B and Qwen3-32B models, while matching or surpassing their accuracy on various TTS benchmarks (AIME-24/25, GPQA-Diamond, MATH-500).
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。