arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.12463cs.CVeess.IV
收藏

LFMamba: 基于状态空间模型的光场图像超分辨率

LFMamba: Light Field Image Super-Resolution with State Space Model

School of Computer, Beijing Institute of Technology(北京理工大学计算机学院);Shenzhen MSU-BIT University(深圳北理莫斯科大学);School of Electronic and Computer Engineering, Shenzhen Graduate School, Peking University(北京大学深圳研究生院电子与计算机工程学院)

Wang xia, Yao Lu, Shunzhou Wang, Ziqi Wang, Peiqi Xia, Tianfei Zhou

AI总结:

针对现有光场超分辨率方法的长距离依赖捕获或计算复杂度问题,提出基于状态空间模型的LFMamba网络,通过在4D光场2D切片上用高效SS2D机制建模特征,在基准测试中取得更优性能。

摘要:

近年来,随着现代神经网络的发展,光场图像超分辨率(LFSR)取得了显著进展。然而,这些方法往往面临(基于CNN的方法)难以捕获长距离依赖,或(基于Transformer的方法)存在二次计算复杂度的挑战,限制了其性能。近期,以Mamba为代表的带有选择性扫描机制(S6)的状态空间模型(SSM),凭借其有效的长距离序列建模能力和线性时间复杂度,在各类视觉任务中成为优于传统基于CNN和Transformer方法的替代方案。因此,将S6集成到LFSR中极具吸引力,尤其是考虑到4D光场的庞大数据量。但主要挑战在于“为4D光场设计合适的扫描方法,以有效建模光场特征”。为解决这一问题,我们在4D光场的信息丰富的2D切片上应用SSM,以充分挖掘空间上下文信息、互补角度信息和结构信息。为此,我们精心设计了一个基础SSM模块,其特点是采用高效的SS2D机制,可在这些2D切片上实现更有效、更高效的特征学习。基于上述两个设计,我们进一步提出了一种用于LFSR的基于SSM的网络,名为LFMamba。在光场基准数据集上的实验结果证明了LFMamba的优越性能。此外,我们开展了大量消融研究,以验证所提方法的有效性和泛化能力。我们期望LFMamba能为利用状态空间模型进行光场的有效表示学习提供思路。

英文摘要:

Recent years have witnessed significant advancements in light field image super-resolution (LFSR) owing to the progress of modern neural networks. However, these methods often face challenges in capturing long-range dependencies (CNN-based) or encounter quadratic computational complexities (Transformer-based), which limit their performance. Recently, the State Space Model (SSM) with selective scanning mechanism (S6), exemplified by Mamba, has emerged as a superior alternative in various vision tasks compared to traditional CNN- and Transformer-based approaches, benefiting from its effective long-range sequence modeling capability and linear-time complexity. Therefore, integrating S6 into LFSR becomes compelling, especially considering the vast data volume of 4D light fields. However, the primary challenge lies in \emph{designing an appropriate scanning method for 4D light fields that effectively models light field features}. To tackle this, we employ SSMs on the informative 2D slices of 4D LFs to fully explore spatial contextual information, complementary angular information, and structure information. To achieve this, we carefully devise a basic SSM block characterized by an efficient SS2D mechanism that facilitates more effective and efficient feature learning on these 2D slices. Based on the above two designs, we further introduce an SSM-based network for LFSR termed LFMamba. Experimental results on LF benchmarks demonstrate the superior performance of LFMamba. Furthermore, extensive ablation studies are conducted to validate the efficacy and generalization ability of our proposed method. We expect that our LFMamba shed light on effective representation learning of LFs with state space models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑