arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2412.07481cs.CV
收藏

Manta: 通过Mamba提升长子序列少样本动作识别

Manta: Enhancing Mamba for Few-Shot Action Recognition of Long Sub-Sequence

Wenbo Huang, Jinghui Zhang, Guang Li, Lei Zhang, Shuoyuan Wang, Fang Dong, Jiahui Jin, Takahiro Ogawa, Miki Haseyama

AI总结:

Manta通过Matryoshka Mamba和混合对比学习框架提升长子序列少样本动作识别,解决传统方法计算复杂度高和局部特征对齐不足的问题,实现SSv2、Kinetics等基准的新SOTA性能。

摘要:

在少样本动作识别(FSAR)中,长视频子序列能更有效地表达完整动作。然而,主流基于Transformer的方法计算复杂度高限制了应用。最近的Mamba在建模长序列上表现出高效性,但直接应用于FSAR忽视了局部特征建模和对齐的重要性。此外,同一类内长子序列积累的类内方差会负面影响FSAR性能。为解决这些挑战,我们提出Matryoshka Mamba和CoNtrasTive LeArning框架(Manta)。首先,Matryoshka Mamba引入多个Inner模块以增强局部特征表示,而非直接建模全局特征。Outer模块捕获这些局部特征之间的时间线依赖性以实现隐式时间对齐。其次,设计了一种混合对比学习范式,结合监督和非监督方法,以缓解类内方差积累的负面影响。Matryoshka Mamba和混合对比学习范式在Manta的两个并行分支中协同工作,提升Mamba在长子序列FSAR中的性能。Manta在SSv2、Kinetics、UCF101和HMDB51等基准上实现了新的SOTA性能。广泛的实证研究证明,Manta从多个角度显著提升了长子序列FSAR性能。

英文摘要:

In few-shot action recognition (FSAR), long sub-sequences of video naturally express entire actions more effectively. However, the high computational complexity of mainstream Transformer-based methods limits their application. Recent Mamba demonstrates efficiency in modeling long sequences, but directly applying Mamba to FSAR overlooks the importance of local feature modeling and alignment. Moreover, long sub-sequences within the same class accumulate intra-class variance, which adversely impacts FSAR performance. To solve these challenges, we propose a Matryoshka MAmba and CoNtrasTive LeArning framework (Manta). Firstly, the Matryoshka Mamba introduces multiple Inner Modules to enhance local feature representation, rather than directly modeling global features. An Outer Module captures dependencies of timeline between these local features for implicit temporal alignment. Secondly, a hybrid contrastive learning paradigm, combining both supervised and unsupervised methods, is designed to mitigate the negative effects of intra-class variance accumulation. The Matryoshka Mamba and the hybrid contrastive learning paradigm operate in two parallel branches within Manta, enhancing Mamba for FSAR of long sub-sequence. Manta achieves new state-of-the-art performance on prominent benchmarks, including SSv2, Kinetics, UCF101, and HMDB51. Extensive empirical studies prove that Manta significantly improves FSAR of long sub-sequence from multiple perspectives.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑