arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2505.14014cs.CV
收藏

EGFormer:面向高效且泛化的多模态语义分割

EGFormer: Towards Efficient and Generalizable Multimodal Semantic Segmentation

University of Sydney(悉尼大学);University of Technology Sydney(悉尼技术大学);HKUST(GZ)(香港科技大学(广州))

Zelin Zhang, Tao Zhang, KediLI, Xu Zheng

AI总结:

针对现有多模态语义分割方法计算效率不足的问题,提出含ASM和MDM模块的EGFormer框架,可灵活整合多模态、降参提效,在合成到真实迁移任务中实现SOTA泛化性能。

摘要:

近期研究已探索采用多种骨干架构实现多模态语义分割。然而,尽管大多数方法旨在提升精度,其计算效率仍未得到充分研究。为解决这一问题,我们提出EGFormer——一种高效的多模态语义分割框架,可灵活整合任意数量的模态,同时在不牺牲性能的前提下显著减少模型参数与推理时间。该框架引入两个全新模块:其一为任意模态评分模块(Any-modal Scoring Module, ASM),独立为每个模态分配重要性分数,实现基于特征图的动态排序;其二为模态丢弃模块(Modal Dropping Module, MDM),在每个阶段过滤掉信息量较低的模态,仅选择性保留并聚合最有价值的特征。这种设计使模型能够利用所有可用模态的有效信息,同时剔除冗余,从而保证高分割质量。除效率外,我们还在合成到真实的迁移任务上评估EGFormer,以验证其泛化能力。大量实验表明,EGFormer取得了具有竞争力的性能,参数减少高达88%,GFLOPs减少50%。在无监督域适应设置下,与现有方法相比,它进一步取得了最优的迁移性能。

英文摘要:

Recent efforts have explored multimodal semantic segmentation using various backbone architectures. However, while most methods aim to improve accuracy, their computational efficiency remains underexplored. To address this, we propose EGFormer, an efficient multimodal semantic segmentation framework that flexibly integrates an arbitrary number of modalities while significantly reducing model parameters and inference time without sacrificing performance. Our framework introduces two novel modules. First, the Any-modal Scoring Module (ASM) assigns importance scores to each modality independently, enabling dynamic ranking based on their feature maps. Second, the Modal Dropping Module (MDM) filters out less informative modalities at each stage, selectively preserving and aggregating only the most valuable features. This design allows the model to leverage useful information from all available modalities while discarding redundancy, thus ensuring high segmentation quality. In addition to efficiency, we evaluate EGFormer on a synthetic-to-real transfer task to demonstrate its generalizability. Extensive experiments show that EGFormer achieves competitive performance with up to 88 percent reduction in parameters and 50 percent fewer GFLOPs. Under unsupervised domain adaptation settings, it further achieves state-of-the-art transfer performance compared to existing methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑