MHA2MLA-VLM: 使DeepSeek的经济型多头潜在注意力在视觉-语言模型中生效
MHA2MLA-VLM: Enabling DeepSeek's Economical Multi-Head Latent Attention across Vision-Language Models
AI总结：
MHA2MLA-VLM通过参数高效的方法将现有视觉-语言模型转换为多头潜在注意力架构，减少缓存足迹并提升推理效率。
摘要：
随着视觉-语言模型（VLMs）处理日益复杂和多模态的任务，快速增长的键值（KV）缓存对推理过程中的内存和计算瓶颈提出了重大挑战。尽管多头潜在注意力（MLA）提供了一种有效的方法来压缩KV缓存并加速推理，但将现有VLMs适应到MLA架构而无需昂贵的预训练仍鲜有探索。在本文中，我们提出了MHA2MLA-VLM，一种参数高效且多模态感知的框架，用于将现成的VLMs转换为MLA。我们的方法包含两个核心技术：（1）一种模态自适应的部分RoPE策略，通过选择性地屏蔽非必要维度，支持传统和多模态设置；（2）一种模态解耦的低秩近似方法，独立压缩视觉和文本的KV空间。此外，我们引入了参数高效的微调来最小化适应成本，并证明了最小化输出激活误差而非参数距离可以显著减少性能损失。在三个代表性VLMs上的广泛实验表明，MHA2MLA-VLM在最小监督数据下恢复原始模型性能，显著减少KV缓存足迹，并与KV量化无缝集成。
英文摘要：
As vision-language models (VLMs) tackle increasingly complex and multimodal tasks, the rapid growth of Key-Value (KV) cache imposes significant memory and computational bottlenecks during inference. While Multi-Head Latent Attention (MLA) offers an effective means to compress the KV cache and accelerate inference, adapting existing VLMs to the MLA architecture without costly pretraining remains largely unexplored. In this work, we present MHA2MLA-VLM, a parameter-efficient and multimodal-aware framework for converting off-the-shelf VLMs to MLA. Our approach features two core techniques: (1) a modality-adaptive partial-RoPE strategy that supports both traditional and multimodal settings by selectively masking nonessential dimensions, and (2) a modality-decoupled low-rank approximation method that independently compresses the visual and textual KV spaces. Furthermore, we introduce parameter-efficient fine-tuning to minimize adaptation cost and demonstrate that minimizing output activation error, rather than parameter distance, substantially reduces performance loss. Extensive experiments on three representative VLMs show that MHA2MLA-VLM restores original model performance with minimal supervised data, significantly reduces KV cache footprint, and integrates seamlessly with KV quantization.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
