撼动视觉语言模型：比较 Transformer 与结构化状态空间模型在视觉与语言建模中的表现
Shaking Up VLMs: Comparing Transformers and Structured State Space Models for Vision & Language Modeling
AI总结：
本文比较了基于 Mamba 与 Transformer 的视觉语言模型，发现 Mamba 在图像描述、问答和阅读理解上更优，而 Transformer 在视觉定位及上下文多模态检索上更强，并分析了视觉编码与检索机制对性能差距的影响。
摘要：
本研究探索用 Mamba（一种近期提出的结构化状态空间模型，在序列建模中展现出良好性能）替代视觉语言模型（VLM）中的 Transformer。我们在受控条件下测试了参数量最高达 3B 的模型，结果表明基于 Mamba 的 VLM 在图像描述、问答和阅读理解任务上优于基于 Transformer 的 VLM。然而，我们发现 Transformer 在视觉定位上取得了更高的性能，且性能差距随规模扩大而加大。我们探索了两种假设来解释这一现象：1）任务无关的视觉编码对隐藏状态更新的影响；2）从上下文多模态检索的角度来看执行视觉定位的困难。我们的结果表明，任务感知编码在定位任务上带来的性能提升很小，而 Transformer 在上下文多模态检索上显著优于 Mamba。总体而言，Mamba 在正确输出依赖于图像整体概括的任务上表现出良好前景，但在需要从上下文中检索显式信息时表现困难。
英文摘要：
This study explores replacing Transformers in Visual Language Models (VLMs) with Mamba, a recent structured state space model (SSM) that demonstrates promising performance in sequence modeling. We test models up to 3B parameters under controlled conditions, showing that Mamba-based VLMs outperforms Transformers-based VLMs in captioning, question answering, and reading comprehension. However, we find that Transformers achieve greater performance in visual grounding and the performance gap widens with scale. We explore two hypotheses to explain this phenomenon: 1) the effect of task-agnostic visual encoding on the updates of the hidden states, and 2) the difficulty in performing visual grounding from the perspective of in-context multimodal retrieval. Our results indicate that a task-aware encoding yields minimal performance gains on grounding, however, Transformers significantly outperform Mamba at in-context multimodal retrieval. Overall, Mamba shows promising performance on tasks where the correct output relies on a summary of the image but struggles when retrieval of explicit information from the context is required.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
