ABMAMBA: 多模态大语言模型中的对齐层次双向扫描用于高效的视频描述
ABMAMBA: Multimodal Large Language Model with Aligned Hierarchical Bidirectional Scan for Efficient Video Captioning
AI总结：
本文提出ABMamba，一种具有线性计算复杂度的多模态大语言模型，通过替代二次注意力机制，实现视频序列的高效处理，在视频描述任务中表现出色。
摘要：
本文研究了通过完全开放的多模态大语言模型（MLLMs）进行视频描述。由于视觉序列的复杂时间依赖性和大量序列长度，现有基于Transformer的方法核心注意力机制随序列长度平方增长，导致计算不可行。为解决这些限制，我们提出了对齐层次双向扫描Mamba（ABMamba），一种具有线性计算复杂度的完全开放的MLLM，能够可扩展地处理视频序列。ABMamba扩展了深度状态空间模型作为其语言骨干，替换了昂贵的二次注意力机制，并采用新颖的对齐层次双向扫描模块，跨多个时间分辨率处理视频。在标准视频描述基准如VATEX和MSR-VTT上，ABMamba在与典型MLLMs相比具有竞争力的同时，实现了约三倍的吞吐量。
英文摘要：
In this study, we focus on video captioning by fully open multimodal large language models (MLLMs). The comprehension of visual sequences is challenging because of their intricate temporal dependencies and substantial sequence length. The core attention mechanisms of existing Transformer-based approaches scale quadratically with the sequence length, making them computationally prohibitive. To address these limitations, we propose Aligned Hierarchical Bidirectional Scan Mamba (ABMamba), a fully open MLLM with linear computational complexity that enables the scalable processing of video sequences. ABMamba extends Deep State Space Models as its language backbone, replacing the costly quadratic attention mechanisms, and employs a novel Aligned Hierarchical Bidirectional Scan module that processes videos across multiple temporal resolutions. On standard video captioning benchmarks such as VATEX and MSR-VTT, ABMamba demonstrates competitive performance compared to typical MLLMs while achieving approximately three times higher throughput.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
