arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2501.04486cs.CV

MB-TaylorFormer V2：通过泰勒公式扩展的改进型多分支线性Transformer用于图像恢复

MB-TaylorFormer V2: Improved Multi-branch Linear Transformer Expanded by Taylor Formula for Image Restoration

School of Intelligent Systems Engineering, Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区智能系统工程学院）；Guangdong Provincial Key Laboratory of Fire Science and Technology（广东省火灾科学与技术重点实验室）；School of Computer Science and Technology, Shenzhen Campus of Harbin Institute of Technology（哈尔滨工业大学深圳校区计算机科学与技术学院）；School of Engineering and Computer Science, Australian National University（澳大利亚国立大学工程与计算机科学学院）；Hong Kong University of Science and Technology（香港科技大学）

Zhi Jin, Yuwei Qiu, Kaihao Zhang, Hongdong Li, Wenhan Luo

备注：
accepted by IEEE TPAMI
阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对Transformer在图像恢复中因Softmax注意力二次复杂度受限的问题，提出MB-TaylorFormer V2，通过泰勒展开近似注意力并引入多分支多尺度架构，实现线性复杂度，在多任务中取得SOTA性能且计算开销小。

摘要：

近年来，Transformer网络凭借全局感受野和对输入的适应性，在图像恢复领域展现出卓越性能。然而，Softmax注意力的二次计算复杂度对其在图像恢复任务中的广泛应用构成了显著限制，尤其是在高分辨率图像上。为应对这一挑战，我们提出了一种新颖的Transformer变体。该变体利用泰勒展开来近似Softmax注意力，并采用保范映射的概念来近似一阶泰勒展开的余项，从而实现线性计算复杂度。此外，我们引入了一种具有多尺度补丁嵌入的多分支架构，该架构具有四个显著优势：1）多种感受野大小；2）多级语义信息；3）灵活的感受野形状；4）加速的训练和推理速度。因此，所提出的模型——基于泰勒公式展开的Transformer第二版（简称MB-TaylorFormer V2）能够同时处理从粗到细的特征，以有限的计算成本捕捉长距离像素交互，并改进泰勒展开余项的近似。在不同图像恢复基准上的实验结果表明，MB-TaylorFormer V2在多种图像恢复任务（如图像去雾、去雨、去雪、运动去模糊和去噪）中实现了最先进的性能，且计算开销极小。源代码可在https://github.com/FVL2020/MB-TaylorFormerV2获取。

英文摘要：

Recently, Transformer networks have demonstrated outstanding performance in the field of image restoration due to the global receptive field and adaptability to input. However, the quadratic computational complexity of Softmax-attention poses a significant limitation on its extensive application in image restoration tasks, particularly for high-resolution images. To tackle this challenge, we propose a novel variant of the Transformer. This variant leverages the Taylor expansion to approximate the Softmax-attention and utilizes the concept of norm-preserving mapping to approximate the remainder of the first-order Taylor expansion, resulting in a linear computational complexity. Moreover, we introduce a multi-branch architecture featuring multi-scale patch embedding into the proposed Transformer, which has four distinct advantages: 1) various sizes of the receptive field; 2) multi-level semantic information; 3) flexible shapes of the receptive field; 4) accelerated training and inference speed. Hence, the proposed model, named the second version of Taylor formula expansion-based Transformer (for short MB-TaylorFormer V2) has the capability to concurrently process coarse-to-fine features, capture long-distance pixel interactions with limited computational cost, and improve the approximation of the Taylor expansion remainder. Experimental results across diverse image restoration benchmarks demonstrate that MB-TaylorFormer V2 achieves state-of-the-art performance in multiple image restoration tasks, such as image dehazing, deraining, desnowing, motion deblurring, and denoising, with very little computational overhead. The source code is available at https://github.com/FVL2020/MB-TaylorFormerV2.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑