arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2403.06977cs.CV
收藏

VideoMamba:用于高效视频理解的状态空间模型

VideoMamba: State Space Model for Efficient Video Understanding

Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);University of Chinese Academy of Sciences(中国科学院大学);State Key Laboratory for Novel Software Technology, Nanjing University(南京大学计算机软件新技术国家重点实验室);OpenGVLab, Shanghai AI Laboratory(上海人工智能实验室 OpenGVLab)

Kunchang Li, Xinhao Li, Yi Wang, Yinan He, Yali Wang, Limin Wang, Yu Qiao

AI总结:

VideoMamba将Mamba状态空间模型适配至视频领域,以线性复杂度解决局部冗余与全局依赖,实现高效长时视频理解,并具备可扩展性、敏感性、优越性和多模态兼容性。

摘要:

针对视频理解中局部冗余和全局依赖的双重挑战,本工作创新性地将Mamba适配到视频领域。所提出的VideoMamba克服了现有3D卷积神经网络和视频Transformer的局限性。其线性复杂度算子能够实现高效的长时建模,这对高分辨率长视频理解至关重要。大量评估揭示了VideoMamba的四个核心能力:(1)在无需大规模数据集预训练的情况下,得益于一种新颖的自蒸馏技术,在视觉领域具有可扩展性;(2)对具有细微运动差异的短期动作识别具有敏感性;(3)在长时视频理解方面具有优越性,相比传统基于特征的模型展现出显著进步;(4)与其他模态的兼容性,在多模态情境中展现出鲁棒性。凭借这些独特优势,VideoMamba为视频理解设立了新基准,为全面视频理解提供了可扩展且高效的解决方案。所有代码和模型可在https://github.com/OpenGVLab/VideoMamba获取。

英文摘要:

Addressing the dual challenges of local redundancy and global dependencies in video understanding, this work innovatively adapts the Mamba to the video domain. The proposed VideoMamba overcomes the limitations of existing 3D convolution neural networks and video transformers. Its linear-complexity operator enables efficient long-term modeling, which is crucial for high-resolution long video understanding. Extensive evaluations reveal VideoMamba's four core abilities: (1) Scalability in the visual domain without extensive dataset pretraining, thanks to a novel self-distillation technique; (2) Sensitivity for recognizing short-term actions even with fine-grained motion differences; (3) Superiority in long-term video understanding, showcasing significant advancements over traditional feature-based models; and (4) Compatibility with other modalities, demonstrating robustness in multi-modal contexts. Through these distinct advantages, VideoMamba sets a new benchmark for video understanding, offering a scalable and efficient solution for comprehensive video understanding. All the code and models are available at https://github.com/OpenGVLab/VideoMamba.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑