arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2606.07571cs.LGcs.AI
收藏

为扩散语言模型启用共享前缀的KV缓存

Enabling KV Caching of Shared Prefix for Diffusion Language Models

Korea University(高丽大学)

Younghun Go, Jaehoon Han, Changyong Shin, Chuck Yoo, Gyeongsik Yang

AI总结:

针对扩散语言模型中双向注意力导致共享前缀KV不稳定的问题,提出双向前缀缓存(bicache),通过动态识别安全层深度重用KV,避免精度崩溃,提升吞吐量36.3%-98.3%。

摘要:

共享前缀的键值(KV)缓存对于高吞吐量的大语言模型(LLM)服务至关重要,但在新兴的扩散语言模型(DLM)中面临严峻挑战。在DLM中,双向注意力意味着更新任何token都会动态改变整个上下文及其对应的KV。因此,为LLM开发的现有缓存技术(假设KV一旦计算就保持不变)会破坏共享前缀KV。我们的实验表明,将这些技术应用于DLM会导致模型精度几乎降为零。为了解锁高吞吐量的DLM服务,我们提出了双向前缀缓存(bicache),这是第一个用于DLM中共享前缀的KV缓存技术。bicache基于我们全面分析的关键观察设计:共享前缀KV在浅层中保持稳定且可重用,而浅层的深度取决于每个请求中共享前缀token的比例。因此,bicache动态识别用于重用共享前缀KV的安全层深度,并消除冗余计算。评估表明,与现有技术相比,bicache显著提高了服务吞吐量36.3%-98.3%,且没有精度崩溃(仅0-1.8%的差异)。

英文摘要:

Key-value (KV) caching for shared prefixes is essential for high-throughput large language model (LLM) serving, but it faces critical challenges in emerging diffusion language models (DLMs). In DLMs, bidirectional attention means that updating any token dynamically alters the entire context and its corresponding KVs. Thus, existing caching techniques developed for LLMs, which assume that KVs remain invariant once computed, corrupt the shared prefix KVs. Our experiments show that applying these techniques to DLMs causes model accuracy to collapse to near zero. To unlock high-throughput DLM serving, we propose bidirectional prefix caching, BiCache, the first KV caching technique for shared prefixes in DLMs. BiCache is designed based on key observations from our comprehensive analysis: shared prefix KVs remain stable and reusable in shallow layers, while the depth of shallow layers depends on the fraction of shared prefix tokens in each request. Thus, BiCache dynamically identifies a safe layer depth for reusing shared prefix KVs and eliminates redundant computation. Evaluations demonstrate that BiCache significantly improves serving throughput by 36.3%-98.3% compared to existing techniques without accuracy collapse (only 0-1.8% difference).

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑