arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.20217cs.LGcs.AI
收藏

KnapSpec: 通过自适应层选择作为背包问题的自推测解码

KnapSpec: Self-Speculative Decoding via Adaptive Layer Selection as a Knapsack Problem

KAIST(韩国科学技术院)

Seongjin Cha, Gyuwan Kim, Dongsu Han, Tao Yang, Insu Han

AI总结:

提出KnapSpec,一种无需训练的框架,将草稿模型选择重新表述为背包问题,通过解耦注意力与MLP层并建模其硬件特定延迟,使用并行动态规划算法自适应确定最优草稿配置,实现令牌吞吐量最大化。

摘要:

自推测解码(SSD)通过跳过层来创建高效的草稿模型,从而加速LLM推理,但现有方法通常依赖静态启发式,忽略了长上下文场景中注意力的动态计算开销。我们提出KnapSpec,一种无需训练的框架,将草稿模型选择重新表述为背包问题,以最大化每时间令牌吞吐量。通过解耦注意力与MLP层,并将其硬件特定延迟建模为上下文长度的函数,KnapSpec通过并行动态规划算法自适应地即时识别最优草稿配置。此外,我们提供了首个严格的理论分析,建立了隐藏状态之间的余弦相似度作为令牌接受率的数学上合理的代理。这一基础使得我们的方法在导航现实世界硬件的动态瓶颈时,能够保持高草稿保真度。我们在Qwen3和Llama3上的实验表明,KnapSpec始终优于最先进的SSD基线,在各种基准测试中实现了高达1.47倍的墙钟加速。我们的即插即用方法确保了长序列的高效推理,无需额外训练或损害目标模型的输出分布。

英文摘要:

Self-speculative decoding (SSD) accelerates LLM inference by skipping layers to create an efficient draft model, yet existing methods often rely on static heuristics that ignore the dynamic computational overhead of attention in long-context scenarios. We propose KnapSpec, a training-free framework that reformulates draft model selection as a knapsack problem to maximize tokens-per-time throughput. By decoupling Attention and MLP layers and modeling their hardware-specific latencies as functions of context length, KnapSpec adaptively identifies optimal draft configurations on the fly via a parallel dynamic programming algorithm. Furthermore, we provide the first rigorous theoretical analysis establishing cosine similarity between hidden states as a mathematically sound proxy for the token acceptance rate. This foundation allows our method to maintain high drafting faithfulness while navigating the shifting bottlenecks of real-world hardware. Our experiments on Qwen3 and Llama3 demonstrate that KnapSpec consistently outperforms state-of-the-art SSD baselines, achieving up to 1.47x wall-clock speedup across various benchmarks. Our plug-and-play approach ensures high-speed inference for long sequences without requiring additional training or compromising the target model's output distribution.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑