arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2611.05304cs.LGcs.AIcs.CL
收藏

LoopVQ:用于循环Transformer的亚1比特KV缓存量化

LoopVQ: Sub 1-bit KV Cache Quantization for Looped Transformers

Seoul National University(首尔大学)

Minsoo Cheong, Sungjoo Yoo

AI总结:

针对循环Transformer的KV缓存线性增长瓶颈,提出LoopVQ联合编码多循环步骤通道并微调码本,在低比特量化下性能优于现有方法,结合TaSQ可在0.25比特保留BF16精度,在SGLang中实现后大幅降低延迟、提升吞吐量。

摘要:

循环Transformer是语言建模中标准Transformer的参数高效替代方案,通过在多个循环步骤中重复使用相同层,近期的循环语言模型可达到规模为其数倍的非循环模型的性能。尽管该架构有效降低了参数内存,但每个循环步骤仍存储自身的键(Key)和值(Value),因此键值(KV)缓存随循环步骤数量线性增长,成为主要内存瓶颈。我们观察到循环模型的KV缓存沿循环轴存在高度冗余,且不同循环步骤对量化的敏感度差异显著。基于这些观察,我们提出LoopVQ——一种用于循环Transformer的KV缓存向量量化方法,该方法联合编码多个循环步骤的通道,并微调码本以匹配全精度模型的输出。在每元素0.5比特的设置下,LoopVQ在通用、长思维链推理及长上下文检索任务中,相较于现有针对标准大语言模型(LLM)的KV缓存向量量化方法,在Ouro和Huginn模型上表现更优。其设计与这些方法正交,结合TaSQ后,即使在每元素0.25比特的设置下,仍能保留大量BF16精度。我们还在SGLang中实现了LoopVQ,在32K token上下文下,其解码延迟几乎减半,与BF16基线相比,在1.4B和2.6B Ouro模型上,峰值服务吞吐量分别提升了最高4.3倍和6.1倍。

英文摘要:

Looped transformers are a parameter-efficient alternative to standard transformers for language modeling. By reapplying the same layers across multiple loop steps, recent looped language models match the performance of non-looped models several times their size. While this architecture effectively reduces parameter memory, each loop step still stores its own keys and values, so the key-value (KV) cache grows linearly with the number of loop steps and becomes the main memory bottleneck. We observe that the KV cache of looped models is highly redundant along the loop axis and that the loop steps differ substantially in their sensitivity to quantization. Based on these observations, we propose $\textbf{LoopVQ}$, a KV cache vector quantization method for looped transformers that codes channels of multiple loop steps jointly, and fine-tunes the codebooks to match the output of the full-precision model. At 0.5 bits per element, LoopVQ outperforms existing KV cache vector quantization methods for standard LLMs on general, long chain-of-thought reasoning, and long-context retrieval tasks with Ouro and Huginn models. Its design is orthogonal to these methods, and combined with TaSQ, it retains much of the BF16 accuracy even at 0.25 bits per element. We also implement LoopVQ in SGLang, where it nearly halves the decode latency at a 32K-token context and raises the peak serving throughput by up to $4.3\times$ and $6.1\times$ on the 1.4B and 2.6B Ouro models compared to the BF16 baseline.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑