VLCache:计算2%的视觉令牌并重用98%用于视觉-语言推断
VLCache: Computing 2% Vision Tokens and Reusing 98% for Vision-Language Inference
AI总结:
VLCache通过重用98%的缓存减少计算量,实现高效视觉-语言推理,精度与全重新计算相当,加速1.2x至16倍。
摘要:
本文提出了VLCache,一种缓存重用框架,利用先前多模态输入中的键值(KV)缓存和编码器缓存,以在相同多模态输入重复时消除昂贵的重新计算。与之前启发式方法不同,我们正式识别了累积重用误差效应,并展示了如何有效最小化非前缀缓存重用误差。我们进一步分析了模型层的重要性变化,并提出了一种动态、层感知的重新计算策略,以在精度和效率之间取得平衡。实验结果表明,VLCache在精度上与完全重新计算相当,同时仅需计算2-5%的令牌,从而获得1.2x至16倍的TTFT加速。我们基于SGLang开发了所提VLCache流水线的实验实现,使实际部署中的推理显著加快。
英文摘要:
This paper presents VLCache, a cache reuse framework that exploits both Key-Value (KV) cache and encoder cache from prior multimodal inputs to eliminate costly recomputation when the same multimodal inputs recur. Unlike previous heuristic approaches, we formally identify the cumulative reuse error effect and demonstrate how to minimize the non-prefix cache reuse error effectively. We further analyze the varying importance of model layers and propose a dynamic, layer-aware recomputation strategy to balance accuracy and efficiency. Experimental results show that VLCache achieves an accuracy on par with full recomputation, while requiring only 2-5% of the tokens to compute, yielding 1.2x-16x TTFT speedups. We develop an experimental implementation of the proposed VLCache pipeline based on SGLang, enabling significantly faster inference in practical deployments.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。