arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.16807cs.LGcs.AI
收藏

通过首层值头跳跃连接改进模型表示并减少KV缓存

Improving Model Representation and Reducing KV Cache via Skip Connections with First Value Heads

Zhoutong Wu, Yuan Zhang, Yiming Dong, Chenheng Zhang, Cong Fang, Kun Yuan, Zhouchen Lin

AI总结:

提出SkipV1Former,通过复用首层值头的跳跃连接,在减少约25%KV缓存的同时提升Transformer表示能力,并支持低成本升级训练及与先进方法结合。

摘要:

Transformer模型凭借其强大的上下文表示学习能力,在各种语言任务中取得了突破性进展。然而,扩展模型规模以提升表示能力往往需要大量的内存和计算成本,例如自回归解码过程中使用的键值(KV)缓存。跳跃连接为在不增加资源消耗的情况下改进表示提供了一种有前景的方法,但大多数先前的工作要么在保持KV成本不变的同时提升表达能力,要么以减少内存为代价而牺牲表示能力。在这项工作中,我们提出了SkipV1Former,一种Transformer变体,它利用来自第一层值头的跳跃连接来增强模型表示并减少KV缓存。具体而言,从第二个块开始,每一层都复用其一半来自最首层的值头,而其余一半则照常计算——从而将值投影和V缓存减少近50%。理论上,我们证明将未压缩的首层值路由到更深层可以恢复因压缩而丢失的信息,并加速模型隐式的内层优化(mesa-optimization)——这是Transformer在自回归任务中的一个关键模式。实验上,在不同的模型规模下,SkipV1Former在相对于标准多头注意力(MHA)Transformer及一些先进变体改进困惑度的同时,一致地实现了约25%的KV缓存减少。此外,我们提出了一种将现有MHA Transformer检查点升级训练(uptraining)为SkipV1Former的方案,仅需额外10-15%的计算量。最后,SkipV1Former可以无缝结合诸如分组查询注意力(Group-Query Attention)和多潜变量注意力(Multi-Latent Attention)等先进方法,以进一步节省KV缓存并提升性能。当与YOCO结合时,它在提升性能的同时将KV缓存大小削减了近50%。

英文摘要:

Transformer models have driven breakthroughs across various language tasks by their strong capability to learn rich contextual representations. Scaling them to improve representation, however, often demands substantial memory and compute costs, such as the Key-Value (KV) cache used during auto-regressive decoding. Skip connections offer a promising way to improve representation without bloating resource usage, yet most prior works either improve expressivity while leaving KV costs unchanged, or reduce memory at the cost of weaker representation. In this work, we propose SkipV1Former, a Transformer variant that uses skip connections from the first layer's Value heads to strengthen model representation and reduce KV cache. Specifically, from the second block onward, each layer reuses half of its Value heads from the very first layer, while computing the other half as usual-cutting Value projections and V cache by nearly 50 \%. Theoretically, we show that routing uncompressed first-layer Values into deeper layers restores information lost to compression and accelerates the model's implicit mesa-optimization-a key pattern of Transformer in auto-regressive tasks. Empirically, across different model scales, SkipV1Former delivers consistent reductions of approximately 25 \% in KV cache while improving perplexity relative to standard Multi-Head Attention (MHA) Transformers and some advanced variants. Moreover, we propose a recipe for uptraining existing MHA Transformer checkpoints to SkipV1Former with only 10-15\% additional compute. Finally, SkipV1Former can seamlessly combine advanced methods like Group-Query Attention and Multi-Latent Attention to achieve further KV cache savings and performance improvement. When combined with YOCO, it cuts KV cache size by nearly 50 \% while still improving performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑