MEDA：面向高效多模态长上下文推理的动态KV缓存分配
MEDA: Dynamic KV Cache Allocation for Efficient Multimodal Long-Context Inference
AI总结：
针对多模态长上下文推理中KV缓存随输入增长导致效率低下的问题，本文提出MEDA方法，利用跨模态注意力熵动态分配各层缓存并合并KV对，实现最高72%内存降低和2.82倍解码加速。
摘要：
融合长文本-图像和文本-视频模态的长上下文多模态大语言模型（MLLMs）随着输入长度增加，其多模态键值（KV）缓存不断增长，需要大量资源，对推理效率构成挑战。现有的KV缓存压缩方法，无论是纯文本还是多模态大语言模型，都忽视了各层之间注意力密度的差异，因此通常采用均匀或渐进缩减策略进行逐层缓存分配。在本工作中，我们提出MEDA，一种动态逐层KV缓存分配方法，用于高效的多模态长上下文推理。其核心是，MEDA利用跨模态注意力熵来确定每个MLLM层的KV缓存大小。在给定每层动态分配的KV缓存大小后，MEDA还采用一种KV对选择方案来识别要选择哪些KV对，以及一种KV对合并策略，将选中的和未选中的KV对合并，以保留整个上下文中的信息。MEDA实现了高达72%的KV缓存内存减少和2.82倍的解码速度提升，同时在长上下文设置下的各种多模态任务中保持或提升性能，包括多图像和长视频场景。我们的代码发布于https://github.com/AIoT-MLSys-Lab/MEDA。
英文摘要：
Long-context Multimodal Large Language Models (MLLMs) that incorporate long text-image and text-video modalities, demand substantial resources as their multimodal Key-Value (KV) caches grow with increasing input lengths, challenging inference efficiency. Existing methods for KV cache compression, in both text-only and multimodal LLMs, have neglected attention density variations across layers, thus often adopting uniform or progressive reduction strategies for layer-wise cache allocation. In this work, we propose MEDA, a dynamic layer-wise KV cache allocation method for efficient multimodal long-context inference. As its core, MEDA utilizes cross-modal attention entropy to determine the KV cache size at each MLLMs layer. Given the dynamically allocated KV cache size at each layer, MEDA also employs a KV pair selection scheme to identify which KV pairs to select and a KV pair merging strategy that merges the selected and non-selected ones to preserve information from the entire context. MEDA achieves up to 72% KV cache memory reduction and 2.82 times faster decoding speed, while maintaining or enhancing performance on various multimodal tasks in long-context settings, including multi-images and long-video scenarios. Our code is released at https://github.com/AIoT-MLSys-Lab/MEDA.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
