Sparrow: 一种基于文本锚定窗口注意力与视觉语义窥视的推测解码方法用于视频大语言模型
Sparrow: Text-Anchored Window Attention with Visual-Semantic Glimpsing for Speculative Decoding in Video LLMs
AI总结:
Sparrow通过文本锚定窗口注意力与视觉语义窥视方法,解决视频大语言模型中推测解码的性能下降问题,实现2.82倍加速。
摘要:
尽管推测解码被广泛用于加速视觉语言模型(VLMs)的推理,但将其应用于视频大语言模型(Vid-LLMs)时会面临严重的性能崩溃。草案模型通常会陷入注意力稀释和负视觉收益的陷阱,这是由于键值缓存爆炸和上下文窗口不匹配所致。我们观察到Vid-LLMs中存在视觉语义内化现象,表明关键视觉语义在深度层交互过程中隐式地编码到文本隐藏状态中,这使得原始视觉输入在深度推理中结构上冗余。为了解决这一问题,我们提出了Sparrow框架,首先利用具有视觉意识的文本锚定窗口注意力,通过隐藏状态重用将所有视觉计算卸载到目标模型上,并利用中间层视觉状态桥接来训练草案模型,使其能够利用富含语义的中间状态,从而过滤掉低层次的视觉噪声。此外,还引入了一种多令牌预测策略以弥合训练-推理分布偏移。实验表明,即使在使用25000个视觉令牌时,Sparrow也实现了2.82倍的平均加速率,有效解决了长序列中的性能下降问题,并为实时长视频任务提供了实用的解决方案。
英文摘要:
Although speculative decoding is widely used to accelerate Vision-Language Models (VLMs) inference, it faces severe performance collapse when applied to Video Large Language Models (Vid-LLMs). The draft model typically falls into the trap of attention dilution and negative visual gain due to key-value cache explosion and context window mismatches. We observe a visual semantic internalization phenomenon in Vid-LLMs, indicating that critical visual semantics are implicitly encoded into text hidden states during deep-layer interactions, which renders raw visual inputs structurally redundant during deep inference. To address this, we propose the Sparrow framework, which first utilizes visually-aware text-anchored window attention via hidden state reuse to fully offload visual computation to the target model, and leverages intermediate-layer visual state bridging to train the draft model with semantic-rich intermediate states, thereby filtering out low-level visual noise. Additionally, a multi-token prediction strategy is introduced to bridge the training-inference distribution shift. Experiments show that Sparrow achieves an average speedup of 2.82x even with 25k visual tokens, effectively resolving the performance degradation in long sequences and offering a practical solution for real-time long video tasks.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。