arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.15091cs.CV

Spatial-Mamba:通过结构感知状态融合实现高效视觉状态空间模型

Spatial-Mamba: Effective Visual State Space Models via Structure-aware State Fusion

The Hong Kong Polytechnic University(香港理工大学);OPPO Research Institute(OPPO研究院);Harvard Medical School(哈佛医学院)

Chaodong Xiao, Minghan Li, Zhengqiang Zhang, Deyu Meng, Lei Zhang

AI总结:

提出Spatial-Mamba,通过在状态空间中引入结构感知状态融合(利用膨胀卷积)直接建立邻域连接,统一Mamba与线性注意力,以单次扫描在图像分类、检测和分割上达到或超越最先进SSM模型。

摘要:

选择性状态空间模型(SSMs),如Mamba,在捕获一维序列数据中的长距离依赖方面表现出色,但其在二维视觉任务中的应用仍面临挑战。当前的视觉SSMs通常将图像转换为1D序列,并采用各种扫描模式来整合局部空间依赖。然而,这些方法在有效捕获复杂的图像空间结构方面存在局限,且因扫描路径延长导致计算成本增加。为解决这些局限,我们提出Spatial-Mamba,一种直接在状态空间中建立邻域连接的新方法。我们不单纯依赖顺序状态转换,而是引入结构感知状态融合方程,利用膨胀卷积捕获图像空间结构依赖,显著增强视觉上下文信息的流动。Spatial-Mamba分三个阶段进行:单向扫描中的初始状态计算、通过结构感知状态融合获取空间上下文、以及利用观测方程进行最终状态计算。我们的理论分析表明,Spatial-Mamba在相同的矩阵乘法框架下统一了原始Mamba和线性注意力,提供了对我们方法的更深入理解。实验结果表明,即使仅使用单次扫描,Spatial-Mamba在图像分类、检测和分割方面也能达到或超越基于SSM的最先进模型。源代码和训练模型可在https://github.com/EdwardChasel/Spatial-Mamba找到。

英文摘要:

Selective state space models (SSMs), such as Mamba, highly excel at capturing long-range dependencies in 1D sequential data, while their applications to 2D vision tasks still face challenges. Current visual SSMs often convert images into 1D sequences and employ various scanning patterns to incorporate local spatial dependencies. However, these methods are limited in effectively capturing the complex image spatial structures and the increased computational cost caused by the lengthened scanning paths. To address these limitations, we propose Spatial-Mamba, a novel approach that establishes neighborhood connectivity directly in the state space. Instead of relying solely on sequential state transitions, we introduce a structure-aware state fusion equation, which leverages dilated convolutions to capture image spatial structural dependencies, significantly enhancing the flow of visual contextual information. Spatial-Mamba proceeds in three stages: initial state computation in a unidirectional scan, spatial context acquisition through structure-aware state fusion, and final state computation using the observation equation. Our theoretical analysis shows that Spatial-Mamba unifies the original Mamba and linear attention under the same matrix multiplication framework, providing a deeper understanding of our method. Experimental results demonstrate that Spatial-Mamba, even with a single scan, attains or surpasses the state-of-the-art SSM-based models in image classification, detection and segmentation. Source codes and trained models can be found at https://github.com/EdwardChasel/Spatial-Mamba.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑