arXiv 2509.118172025-09-15cs.CV

MAFS：用于红外与可见光图像融合及语义分割的掩码自编码器

MAFS: Masked Autoencoder for Infrared-Visible Image Fusion and Semantic Segmentation

Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Jilin University（教育部符号计算与知识工程重点实验室，吉林大学）；Wangxuan Institute of Computer Technology, Peking University（北京大学王轩计算机技术研究所）；National Engineering Research Center of Visual Technology, School of Computer Science, Peking University（视觉技术国家工程研究中心，北京大学计算机科学学院）

Liying Wang, Xiaoli Zhang, Chuanmin Jia, Siwei Ma

备注：: Accepted by TIP 2025

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出统一网络MAFS，通过并行融合与分割子网络、异构特征融合策略及多阶段Transformer解码器，实现红外与可见光图像融合与语义分割的相互促进，取得有竞争力的结果。

摘要：

红外与可见光图像融合方法旨在生成具有良好视觉质量的融合图像，并促进高级任务的性能。事实上，现有的语义驱动方法已考虑为下游应用注入语义信息。然而，它们均未从宏观任务级视角探究逐像素图像融合与跨模态特征融合感知任务之间相互促进的潜力。为解决此局限，我们提出一个用于图像融合和语义分割的统一网络。MAFS采用并行结构，包含融合子网络和分割子网络。一方面，我们设计了异构特征融合策略，以增强图像融合的语义感知能力。另一方面，通过级联融合子网络和分割骨干网络，转移分割相关知识以促进基于特征级融合的分割。在该框架内，我们设计了一种新颖的多阶段Transformer解码器，以高效聚合细粒度的多尺度融合特征。此外，引入基于最大-最小公平分配原则的动态因子，生成两项任务的自适应权重，并保证以多任务方式进行平滑训练。大量实验表明，与最先进方法相比，我们的方法取得了有竞争力的结果。代码可在https://github.com/Abraham-Einstein/MAFS/获取。

英文摘要：

Infrared-visible image fusion methods aim at generating fused images with good visual quality and also facilitate the performance of high-level tasks. Indeed, existing semantic-driven methods have considered semantic information injection for downstream applications. However, none of them investigates the potential for reciprocal promotion between pixel-wise image fusion and cross-modal feature fusion perception tasks from a macroscopic task-level perspective. To address this limitation, we propose a unified network for image fusion and semantic segmentation. MAFS is a parallel structure, containing a fusion sub-network and a segmentation sub-network. On the one hand, We devise a heterogeneous feature fusion strategy to enhance semantic-aware capabilities for image fusion. On the other hand, by cascading the fusion sub-network and a segmentation backbone, segmentation-related knowledge is transferred to promote feature-level fusion-based segmentation. Within the framework, we design a novel multi-stage Transformer decoder to aggregate fine-grained multi-scale fused features efficiently. Additionally, a dynamic factor based on the max-min fairness allocation principle is introduced to generate adaptive weights of two tasks and guarantee smooth training in a multi-task manner. Extensive experiments demonstrate that our approach achieves competitive results compared with state-of-the-art methods. The code is available at https://github.com/Abraham-Einstein/MAFS/.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。