Vamba：使用混合Mamba-Transformer理解小时级长视频
Vamba: Understanding Hour-Long Videos with Hybrid Mamba-Transformers
AI总结：
本文提出混合Mamba-Transformer模型VAMBA，利用Mamba-2块以线性复杂度编码视频令牌，无需令牌压缩即可处理小时级长视频，显著降低GPU内存并提升训练速度，在LVBench基准上准确率提高4.3%。
摘要：
最先进的基于Transformer的大型多模态模型（LMMs）由于因果自注意力操作的二次复杂度，难以处理小时级长视频输入，导致训练和推理期间的计算成本高昂。现有的基于令牌压缩的方法减少了视频令牌的数量，但通常会导致信息损失，并且对于极长序列仍然效率低下。在本文中，我们探索了一个正交方向，构建一个混合Mamba-Transformer模型（VAMBA），该模型采用Mamba-2块以线性复杂度对视频令牌进行编码。在没有任何令牌减少的情况下，VAMBA可以在单个GPU上编码超过1024帧（640×360），而基于Transformer的模型只能编码256帧。对于长视频输入，VAMBA在训练和推理期间的GPU内存使用量至少减少了50%，并且与基于Transformer的LMMs相比，每个训练步骤的速度几乎翻倍。我们的实验结果表明，在具有挑战性的小时级长视频理解基准LVBench上，与先前的高效视频LMMs相比，VAMBA的准确率提高了4.3%，并在广泛的短视频和长视频理解任务中保持了强劲的性能。
英文摘要：
State-of-the-art transformer-based large multimodal models (LMMs) struggle to handle hour-long video inputs due to the quadratic complexity of the causal self-attention operations, leading to high computational costs during training and inference. Existing token compression-based methods reduce the number of video tokens but often incur information loss and remain inefficient for extremely long sequences. In this paper, we explore an orthogonal direction to build a hybrid Mamba-Transformer model (VAMBA) that employs Mamba-2 blocks to encode video tokens with linear complexity. Without any token reduction, VAMBA can encode more than 1024 frames (640$\times$360) on a single GPU, while transformer-based models can only encode 256 frames. On long video input, VAMBA achieves at least 50% reduction in GPU memory usage during training and inference, and nearly doubles the speed per training step compared to transformer-based LMMs. Our experimental results demonstrate that VAMBA improves accuracy by 4.3% on the challenging hour-long video understanding benchmark LVBench over prior efficient video LMMs, and maintains strong performance on a broad spectrum of long and short video understanding tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
