基于Mamba的第一人称视频跨模态动作识别：通过CLS令牌融合策略整合RGB和手部骨架流
Cross-Modal Action Recognition in Egocentric Video Using Mamba: Integrating RGB and Hand Skeleton Streams via CLS Token Fusion Strategies
AI总结：
提出一种基于Mamba的跨模态架构，通过四种CLS令牌融合策略（朴素、平均、加权和基于上下文）整合RGB视频和手部骨架数据，在H2O数据集上平均策略达到最佳性能，Top-1准确率在Tiny配置下提升超10%。
摘要：
第一人称动作识别由于相机运动不稳定、手部频繁遮挡以及随时间保持一致视觉表示的困难而具有挑战性。在这项工作中，我们提出了一种跨模态架构，将RGB视频和时间手部骨架数据结合在一个统一的基于Mamba的框架内，利用状态空间模型（SSMs）的线性时间复杂度。我们的架构由三个组件组成：用于视觉特征提取的VideoMamba模块、基于Mamba块堆叠的骨架编码器，以及将两种模态整合为单一表示的融合模块。本工作的一个核心贡献是设计和评估了四种用于多模态融合的类（CLS）令牌混合策略：朴素、平均、加权和基于上下文。这些策略在如何利用预训练的单模态CLS令牌（其作用是作为信息汇聚集所学表示）来初始化用于最终分类的混合CLS令牌方面有所不同。我们在H2O数据集上评估了所有策略。实验结果表明，平均策略实现了最佳性能，在Tiny配置下比VideoMamba基线提高了超过10%的Top-1准确率，在Small配置下提高了2%。
英文摘要：
Egocentric action recognition is a challenging task due to erratic camera motion, frequent hand occlusion, and the difficulty of maintaining consistent visual representations over time. In this work, we propose a cross-modal architecture that combines RGB video and temporal hand skeleton data within a unified Mamba-based framework, exploiting the linear time complexity of State Space Models (SSMs). Our architecture consists of three components: a VideoMamba module for visual feature extraction, a skeleton encoder built on a stack of Mamba blocks, and a fusion module that integrates both modalities into a single representation. A central contribution of this work is the design and evaluation of four Class (CLS) token mixing strategies for multimodal fusion: Naive, Average, Weighted and Context-based. These strategies differ in how the pretrained unimodal CLS tokens, which role is to act as information sinks concentrating learned representations, are leveraged to initialize the mixed CLS token used for final classification. We evaluate all strategies on the H2O dataset. Experimental results show that the Average strategy achieves the best performance, yielding gains of over 10% Top-1 accuracy in the Tiny configuration and 2% in the Small configuration over the VideoMamba baseline.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
