PrefixKV:自适应前缀KV缓存是视觉指令跟随模型高效生成所需的关键
PrefixKV: Adaptive Prefix KV Cache is What Vision Instruction-Following Models Need for Efficient Generation
AI总结:
针对视觉指令跟随模型推理时KV缓存开销大且各层重要性不均的问题,提出PrefixKV方法,通过二分搜索自适应分配各层KV缓存大小,保留关键上下文,实现高效生成并达到最先进性能。
摘要:
近年来,大型视觉语言模型(LVLMs)因其在多样多模态输入下强大的生成和推理能力而迅速普及。然而,这些模型在推理过程中会产生显著的计算和内存开销,这极大地阻碍了其在实际场景中的高效部署。由长输入和输出序列所必需的大量键值(KV)缓存,显著导致了高推理成本。基于此,近期研究探索了减小KV缓存大小以提高效率的方法。尽管这些方法有效,但它们通常忽略了各层KV向量的重要性分布差异,并在预测下一个token时为每层维持相同的缓存大小。这导致某些层的重要上下文信息丢失,进而引起显著的性能下降。为解决此问题,我们提出了PrefixKV,其中“Prefix”指基于重要性排名而非原始序列位置的KV。它将确定所有层KV缓存大小的挑战重新定义为搜索最优全局前缀配置的任务。通过基于二分搜索的自适应逐层KV保留策略,每层得以保留最大限度的上下文信息,从而促进生成。大量实验表明,与其他方法相比,我们的方法达到了最先进的性能。它在推理效率和生成质量之间展现出优越的权衡,显示出实际应用的巨大潜力。代码可在 https://github.com/THU-MIG/PrefixKV 获取。
英文摘要:
Recently, large vision-language models (LVLMs) have rapidly gained popularity for their strong generation and reasoning capabilities given diverse multimodal inputs. However, these models incur significant computational and memory overhead during inference, which greatly hinders the efficient deployment in practical scenarios. The extensive key-value (KV) cache, necessitated by the lengthy input and output sequences, notably contributes to the high inference cost. Based on this, recent works have investigated ways to reduce the KV cache size for higher efficiency. Although effective, they generally overlook the distinct importance distributions of KV vectors across layers and maintain the same cache size for each layer during the next token prediction. This results in the significant contextual information loss for certain layers, leading to notable performance decline. To address this, we present PrefixKV, where "Prefix" means the top-ranked KV based on importance rather than position in the original sequence. It reframes the challenge of determining KV cache sizes for all layers into the task of searching for the optimal global prefix configuration. With an adaptive layer-wise KV retention recipe based on binary search, the maximum contextual information can thus be preserved in each layer, facilitating the generation. Extensive experiments demonstrate that our method achieves the state-of-the-art performance compared with others. It exhibits superior inference efficiency and generation quality trade-offs, showing promising potential for practical applications. Code is available at https://github.com/THU-MIG/PrefixKV.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。