arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.19006cs.CV
收藏

蛇与梯子:VideoMamba 的两步提升

Snakes and Ladders: Two Steps Up for VideoMamba

Utrecht University(乌得勒支大学)

Hui Lu, Albert Ali Salah, Ronald Poppe

AI总结:

本文提出 VideoMambaPro,通过引入掩码后向计算和元素残差连接解决 Mamba 的历史衰减与元素矛盾问题,在多个视频基准上显著提升性能,成为 transformer 的高效替代方案。

摘要:

视频理解需要提取丰富的时空表征,transformer 模型通过自注意力机制实现这一点。然而,自注意力带来了计算负担。在 NLP 中,Mamba 作为 transformer 的高效替代方案已崭露头角。但是,Mamba 的成功并不能轻易扩展到包括视频分析在内的视觉任务中。本文从理论上分析了自注意力与 Mamba 之间的差异。我们发现了 Mamba 在 token 处理中的两个局限性:历史衰减和元素矛盾。我们提出了 VideoMambaPro (VMP),通过向 VideoMamba 主干网络添加掩码后向计算和元素残差连接来解决所发现的局限性。不同规模的 VideoMambaPro 模型在 Kinetics-400 和 Something-Something V2 上分别以 1.6-2.8% 和 1.1-1.9% 的 top-1 准确率超越了 VideoMamba。即使没有广泛的预训练,我们的模型也日益成为当前 transformer 模型极具吸引力的、高效的替代方案。此外,我们的两项解决方案与 Vision Mamba 模型的最新进展是正交的,并有望在未来的模型中提供进一步的改进。

英文摘要:

Video understanding requires the extraction of rich spatio-temporal representations, which transformer models achieve through self-attention. Unfortunately, self-attention poses a computational burden. In NLP, Mamba has surfaced as an efficient alternative for transformers. However, Mamba's successes do not trivially extend to vision tasks, including those in video analysis. In this paper, we theoretically analyze the differences between self-attention and Mamba. We identify two limitations in Mamba's token processing: historical decay and element contradiction. We propose VideoMambaPro (VMP) that solves the identified limitations by adding masked backward computation and elemental residual connections to a VideoMamba backbone. Differently sized VideoMambaPro models surpass VideoMamba by 1.6-2.8% and 1.1-1.9% top-1 on Kinetics-400 and Something-Something V2, respectively. Even without extensive pre-training, our models present an increasingly attractive and efficient alternative to current transformer models. Moreover, our two solutions are orthogonal to recent advances in Vision Mamba models, and are likely to provide further improvements in future models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑