视觉语言模型需要视觉变换器吗？评估状态空间模型作为视觉编码器
Do VLMs Need Vision Transformers? Evaluating State Space Models as Vision Encoders
AI总结：
本文评估了状态空间模型作为视觉编码器在视觉语言模型中的有效性，发现其在VQA和定位任务中表现优异，并提出稳定策略提升鲁棒性。
摘要：
大型视觉-语言模型（VLMs）通常使用冻结的视觉主干，其图像特征通过轻量级连接器映射到大语言模型。尽管基于变换器的编码器是标准视觉主干，我们询问状态空间模型（SSM）视觉主干是否能成为强大替代方案。我们系统评估了SSM视觉主干在受控环境下的VLMs性能。在匹配的ImageNet-1K初始化下，SSM主干在VQA和定位任务中总体表现最强。我们进一步对SSM和ViT家族主干进行检测或分割训练，并发现密集任务微调在各家族中普遍提升性能；在适应后，SSM主干仍具竞争力，同时模型规模显著更小。我们还发现（i）更高的ImageNet准确率或更大的主干不总是带来更好的VLM性能，（ii）某些视觉主干在定位任务中不稳定。基于这些发现，我们提出稳定策略，提升两种主干的鲁棒性，并强调SSM主干作为变换器视觉编码器在VLMs中的强大替代方案。
英文摘要：
Large vision--language models (VLMs) often use a frozen vision backbone, whose image features are mapped into a large language model through a lightweight connector. While transformer-based encoders are the standard visual backbone, we ask whether state space model (SSM) vision backbones can be a strong alternative. We systematically evaluate SSM vision backbones for VLMs in a controlled setting. Under matched ImageNet-1K initialization, the SSM backbone achieves the strongest overall performance across both VQA and grounding/localization. We further adapt both SSM and ViT-family backbones with detection or segmentation training and find that dense-task tuning generally improves performance across families; after this adaptation, the SSM backbone remains competitive while operating at a substantially smaller model scale. We further observe that (i) higher ImageNet accuracy or larger backbones do not reliably translate into better VLM performance, and (ii) some visual backbones are unstable in localization. Based on these findings, we propose stabilization strategies that improve robustness for both backbone families and highlight SSM backbones as a strong alternative to transformer-based vision encoders in VLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
