EntropyCache: 基于解码令牌熵的KV缓存用于扩散语言模型
EntropyCache: Decoded Token Entropy Guided KV Caching for Diffusion Language Models
AI总结:
EntropyCache利用解码令牌熵指导KV缓存,通过最大熵信号决定是否重新计算,实现高效缓存更新,提升推理速度和效率。
摘要:
扩散式大语言模型(dLLMs)依赖双向注意力,这阻止了无损的KV缓存并要求每次去噪步骤都进行完整前向传递。现有的近似KV缓存方法通过选择性更新缓存状态来减少此成本,但其决策开销随上下文长度或模型深度而增加。我们提出了EntropyCache,一种无需训练的KV缓存方法,利用新解码令牌分布的最大熵作为常数成本信号来决定何时重新计算。我们的设计基于两个经验观察:(1)解码令牌熵与KV缓存漂移相关,提供一个廉价的缓存过时代理;(2)解码令牌的特征波动在取消掩码后持续多个步骤,促使重新计算最近期解码的$k$个令牌。跳过或重新计算的决策仅需每个步骤$O(V)$的计算,与上下文长度和模型规模无关。在LLaDA-8B-Instruct和Dream-7B-Instruct上的实验表明,EntropyCache在标准基准上实现15.2×-26.4×的速度提升,在链式思维基准上实现22.4×-24.1×的速度提升,具有竞争性的准确性和决策开销仅占推理时间的0.5%。代码可在https://github.com/mscheong01/EntropyCache上获得。
英文摘要:
Diffusion-based large language models (dLLMs) rely on bidirectional attention, which prevents lossless KV caching and requires a full forward pass at every denoising step. Existing approximate KV caching methods reduce this cost by selectively updating cached states, but their decision overhead scales with context length or model depth. We propose EntropyCache, a training-free KV caching method that uses the maximum entropy of newly decoded token distributions as a constant-cost signal for deciding when to recompute. Our design is grounded in two empirical observations: (1) decoded token entropy correlates with KV cache drift, providing a cheap proxy for cache staleness, and (2) feature volatility of decoded tokens persists for multiple steps after unmasking, motivating recomputation of the $k$ most recently decoded tokens. The skip-or-recompute decision requires only $O(V)$ computation per step, independent of context length and model scale. Experiments on LLaDA-8B-Instruct and Dream-7B-Instruct show that EntropyCache achieves $15.2\times$-$26.4\times$ speedup on standard benchmarks and $22.4\times$-$24.1\times$ on chain-of-thought benchmarks, with competitive accuracy and decision overhead accounting for only $0.5\%$ of inference time. Code is available at https://github.com/mscheong01/EntropyCache.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。