稀疏到密集：一种无损加速视频理解的LLM免费午餐
Sparse-to-Dense: A Free Lunch for Lossless Acceleration of Video Understanding in LLMs
AI总结：
本文提出了一种名为Sparse-to-Dense（StD）的解码策略，通过结合稀疏top-K注意力和密集全注意力模块，实现视频大语言模型（Video-LLMs）的无损加速，从而在处理长视频序列时显著提高处理速度。
摘要：
由于当前视频大语言模型（Video-LLMs）的自回归性质，输入序列长度的增长会导致推理延迟增加，这给处理通常非常长的视频序列带来了挑战。我们发现，在解码过程中，Video-LLMs中大多数标记的注意力分数趋于稀疏和集中，只有某些标记需要全面的全注意力。基于这一见解，我们引入了Sparse-to-Dense（StD），一种新颖的解码策略，集成了两个不同的模块：一个利用稀疏top-K注意力，另一个采用密集全注意力。这些模块协同工作，以在不损失的情况下加速Video-LLMs。快速（稀疏）模型推测解码多个标记，而缓慢（密集）模型并行验证它们。StD是一种无调优、即插即用的解决方案，可在视频处理中实现高达1.94倍的壁时加速。它在保持模型性能的同时，使从标准Video-LLM无缝过渡到稀疏Video-LLM变得可能，只需最小的代码修改。
英文摘要：
Due to the auto-regressive nature of current video large language models (Video-LLMs), the inference latency increases as the input sequence length grows, posing challenges for the efficient processing of video sequences that are usually very long. We observe that during decoding, the attention scores of most tokens in Video-LLMs tend to be sparse and concentrated, with only certain tokens requiring comprehensive full attention. Based on this insight, we introduce Sparse-to-Dense (StD), a novel decoding strategy that integrates two distinct modules: one leveraging sparse top-K attention and the other employing dense full attention. These modules collaborate to accelerate Video-LLMs without loss. The fast (sparse) model speculatively decodes multiple tokens, while the slow (dense) model verifies them in parallel. StD is a tuning-free, plug-and-play solution that achieves up to a 1.94$\times$ walltime speedup in video processing. It maintains model performance while enabling a seamless transition from a standard Video-LLM to a sparse Video-LLM with minimal code modifications.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
