arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.01901cs.CV
收藏

Q Cache:视觉注意力在少于一半的解码层中具有价值用于多模态大语言模型

Q Cache: Visual Attention is Valuable in Less than Half of Decode Layers for Multimodal Large Language Model

Jiedong Zhuang, Lu Lu, Ming Dai, Rui Hu, Jian Chen, Qiang Liu, Haoji Hu

AI总结:

Q Cache通过跨层共享相似注意力模式,减少多模态大语言模型中的KV缓存使用量,提升吞吐量并保持性能

摘要:

多模态大语言模型(MLLMs)受到视觉编码器中大量视觉标记带来的高昂推理成本的困扰。这些冗余的视觉标记导致了显著的计算负载和关键值(KV)缓存瓶颈。现有方法专注于逐个标记优化,利用多样复杂的标记剪枝技术来消除非关键的视觉标记。然而,这些方法往往不可避免地破坏KV缓存的完整性,导致长文本生成任务失败。为此,我们从新的角度深入研究模型的注意力机制,并发现超过一半的解码层中的注意力是语义相似的。基于这一发现,我们认为某些层中的注意力可以通过继承前一层的注意力进行简化。因此,我们提出了Lazy Attention,一种高效的注意力机制,使相似的注意力模式能够在不同层之间共享。在Lazy Attention中,我们开发了一种新的层共享缓存,Q Cache,专门针对MLLMs,以促进相邻层之间查询的重用。特别是,Q Cache轻量且完全兼容现有的推理框架,包括Flash Attention和KV缓存。此外,我们的方法高度灵活,因为它与现有的逐个标记技术正交,可以独立部署或与标记剪枝方法结合使用。在多个基准上的实证评估表明,我们的方法可以将KV缓存使用量减少超过35%,并实现1.5倍的吞吐量提升,同时在各种MLLMs上仅牺牲约1%的性能。与最先进的逐个标记方法相比,我们的技术实现了更高的准确性保留。

英文摘要:

Multimodal large language models (MLLMs) are plagued by exorbitant inference costs attributable to the profusion of visual tokens within the vision encoder. The redundant visual tokens engenders a substantial computational load and key-value (KV) cache footprint bottleneck. Existing approaches focus on token-wise optimization, leveraging diverse intricate token pruning techniques to eliminate non-crucial visual tokens. Nevertheless, these methods often unavoidably undermine the integrity of the KV cache, resulting in failures in long-text generation tasks. To this end, we conduct an in-depth investigation towards the attention mechanism of the model from a new perspective, and discern that attention within more than half of all decode layers are semantic similar. Upon this finding, we contend that the attention in certain layers can be streamlined by inheriting the attention from their preceding layers. Consequently, we propose Lazy Attention, an efficient attention mechanism that enables cross-layer sharing of similar attention patterns. It ingeniously reduces layer-wise redundant computation in attention. In Lazy Attention, we develop a novel layer-shared cache, Q Cache, tailored for MLLMs, which facilitates the reuse of queries across adjacent layers. In particular, Q Cache is lightweight and fully compatible with existing inference frameworks, including Flash Attention and KV cache. Additionally, our method is highly flexible as it is orthogonal to existing token-wise techniques and can be deployed independently or combined with token pruning approaches. Empirical evaluations on multiple benchmarks demonstrate that our method can reduce KV cache usage by over 35% and achieve 1.5x throughput improvement, while sacrificing only approximately 1% of performance on various MLLMs. Compared with SOTA token-wise methods, our technique achieves superior accuracy preservation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑