通过注意力驱动的自我压缩进行视觉标记减少以提高高效多模态大语言模型的效率
Vision Token Reduction via Attention-Driven Self-Compression for Efficient Multimodal Large Language Models
AI总结：
通过注意力驱动的自我压缩方法，有效减少多模态大语言模型中的视觉标记，提升计算效率并保持模型性能。
摘要：
多模态大语言模型（MLLMs）在处理大量视觉标记时通过所有LLM层会带来显著的计算成本。先前的剪枝方法要么在LLM之前进行，由于编码器-投影器设计的多样性限制了通用性，要么在LLM内部使用启发式方法，这些方法与FlashAttention不兼容。我们采取了不同的方法：而不是识别不重要的标记，我们将LLM本身作为压缩的最佳指南。观察到更深层的层自然地传输视觉到文本信息，我们引入了注意力驱动的自我压缩（ADSC），一种简单且广泛适用的方法，通过仅使用LLM的注意力机制逐步减少视觉标记。我们的方法在选定的层上应用均匀的标记下采样，形成瓶颈，促使模型重新组织和压缩信息到剩余的标记中。它不需要计算分数、辅助模块或修改注意力，且与FlashAttention完全兼容。应用于LLaVA-1.5时，ADSC将FLOPs减少53.7%，峰值KV缓存内存减少56.7%，同时保持98.2%的原始模型性能。在多个基准测试中，它在效率和准确性上都优于先前的剪枝方法。关键的是，在高压缩比下，我们的方法保持稳健，而基于启发式的技巧则急剧下降。
英文摘要：
Multimodal Large Language Models (MLLMs) incur significant computational cost from processing numerous vision tokens through all LLM layers. Prior pruning methods operate either before the LLM, limiting generality due to diverse encoder-projector designs or within the LLM using heuristics that are incompatible with FlashAttention. We take a different approach: rather than identifying unimportant tokens, we treat the LLM itself as the optimal guide for compression. Observing that deeper layers naturally transmit vision-to-text information, we introduce Attention-Driven Self-Compression (ADSC), a simple, broadly applicable method that progressively reduces vision tokens using only the LLM's attention mechanism. Our method applies uniform token downsampling at selected layers, forming bottlenecks that encourage the model to reorganize and compress information into the remaining tokens. It requires no score computation, auxiliary modules, or attention modification, and remains fully compatible with FlashAttention. Applied to LLaVA-1.5, ADSC reduces FLOPs by 53.7% and peak KV-cache memory by 56.7%, while preserving 98.2% of the original model performance. Across multiple benchmarks, it outperforms prior pruning approaches in both efficiency and accuracy. Crucially, under high compression ratios, our method remains robust while heuristic-based techniques degrade sharply.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
