Mamba Fusion：通过提问学习动作
Mamba Fusion: Learning Actions Through Questioning
AI总结：
提出MambaVL模型，利用选择性状态空间模态融合与共享状态转移矩阵高效捕获长程依赖，并结合问答任务引导模型关注相关线索，在Epic-Kitchens-100数据集上取得动作识别和预测的最优性能。
摘要：
Video Language Models (VLMs) 对于在多样化任务中进行泛化以及利用语言线索增强学习至关重要。尽管基于 transformer 的架构一直是视觉语言训练的事实标准，但它们面临着诸如二次计算复杂度、高 GPU 内存使用量以及难以处理长期依赖关系等挑战。为了解决这些局限性，我们引入了 MambaVL，这是一种新颖的模型，它利用选择性状态空间模态融合的最新进展，来高效捕获长程依赖关系并学习视觉和语言数据的联合表示。MambaVL 在两种模态之间利用共享的状态转移矩阵，使模型能够从场景内的多个视角捕获动作信息。此外，我们提出了一项问答任务，有助于引导模型关注相关线索。这些问题提供了关于动作、物体和环境上下文的关键信息，从而提升了性能。因此，MambaVL 在 Epic-Kitchens-100 数据集上的动作识别任务中取得了最先进的性能，并在动作预测任务中优于基线方法。
英文摘要：
Video Language Models (VLMs) are crucial for generalizing across diverse tasks and using language cues to enhance learning. While transformer-based architectures have been the de facto in vision-language training, they face challenges like quadratic computational complexity, high GPU memory usage, and difficulty with long-term dependencies. To address these limitations, we introduce MambaVL, a novel model that leverages recent advancements in selective state space modality fusion to efficiently capture long-range dependencies and learn joint representations for vision and language data. MambaVL utilizes a shared state transition matrix across both modalities, allowing the model to capture information about actions from multiple perspectives within the scene. Furthermore, we propose a question-answering task that helps guide the model toward relevant cues. These questions provide critical information about actions, objects, and environmental context, leading to enhanced performance. As a result, MambaVL achieves state-of-the-art performance in action recognition on the Epic-Kitchens-100 dataset and outperforms baseline methods in action anticipation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
