arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.10316cs.CV
收藏

GlobalMamba:面向 Vision Mamba 的全局图像序列化

GlobalMamba: Global Image Serialization for Vision Mamba

Tsinghua University(清华大学);UC Berkeley(加州大学伯克利分校)

Chengkun Wang, Wenzhao Zheng, Jie Zhou, Jiwen Lu

AI总结:

针对现有 Vision Mamba 忽略图像 2D 结构且难以提取全局信息的问题,提出全局图像序列化方法 GlobalMamba,利用 DCT 将图像转换为包含全局信息的因果 token 序列,在图像分类、目标检测和语义分割任务上验证了其有效性。

摘要:

Vision mambas 在视觉 token 数量方面展现出了线性复杂度的强大性能。其高效性源于对图像 token 的顺序处理。然而,现有大多数方法采用基于图像块的图像 token 化,并将其展平为 1D 序列以进行因果处理,这忽略了图像固有的 2D 结构相关性。通过局部图像块的顺序处理也很难提取全局信息。在本文中,我们提出了一种全局图像序列化方法,将图像转换为包含 2D 图像全局信息的因果 token 序列。我们首先使用 Discrete Cosine Transform (DCT) 将图像从空间域转换到频域,然后按相应的频率范围排列像素。我们进一步将同一频带内的每个集合转换回空间域,以在 token 化之前获得一系列图像。我们基于提出的全局图像序列化构建了具有因果输入格式的 vision mamba 模型 GlobalMamba,该模型能更好地利用图像序列间的因果关系。大量实验证明了 GlobalMamba 的有效性,包括在 ImageNet-1K 上的图像分类、COCO 上的目标检测以及 ADE20K 上的语义分割。

英文摘要:

Vision mambas have demonstrated strong performance with linear complexity to the number of vision tokens. Their efficiency results from processing image tokens sequentially. However, most existing methods employ patch-based image tokenization and then flatten them into 1D sequences for causal processing, which ignore the intrinsic 2D structural correlations of images. It is also difficult to extract global information by sequential processing of local patches. In this paper, we propose a global image serialization method to transform the image into a sequence of causal tokens, which contain global information of the 2D image. We first convert the image from the spatial domain to the frequency domain using Discrete Cosine Transform (DCT) and then arrange the pixels with corresponding frequency ranges. We further transform each set within the same frequency band back to the spatial domain to obtain a series of images before tokenization. We construct a vision mamba model, GlobalMamba, with a causal input format based on the proposed global image serialization, which can better exploit the causal relations among image sequences. Extensive experiments demonstrate the effectiveness of our GlobalMamba, including image classification on ImageNet-1K, object detection on COCO, and semantic segmentation on ADE20K.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑