arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.08613cs.CVcs.AI
收藏

跨模态双向交互模型用于指代遥感图像分割

Cross-Modal Bidirectional Interaction Model for Referring Remote Sensing Image Segmentation

School of Electronics and Information Engineering, Harbin Institute of Technology(电子信息工程学院,哈尔滨工业大学);Faculty of Computing, Harbin Institute of Technology(计算机学院,哈尔滨工业大学);Peng Cheng Laboratory, Shenzhen(鹏城实验室,深圳)

Zhe Dong, Yuzhe Sun, Tianzhu Liu, Wangmeng Zuo, Yanfeng Gu

AI总结:

提出跨模态双向交互模型CroBIM,通过上下文感知提示调制、语言引导特征聚合和互交互解码器解决指代遥感图像分割中复杂空间关系与尺度差异问题,并构建大规模基准RISBench,性能优于现有方法。

摘要:

给定自然语言表达和遥感图像,指代遥感图像分割(RRSIS)的目标是生成由指代表达所标识的目标对象的像素级掩码。与自然场景相比,RRSIS中的表达通常涉及复杂的地理空间关系,目标对象在尺度上差异显著且缺乏视觉显著性,从而增加了实现精确分割的难度。为了解决上述挑战,提出了一种新颖的RRSIS框架,称为跨模态双向交互模型(CroBIM)。具体而言,设计了一个上下文感知提示调制(CAPM)模块,将空间位置关系和任务特定知识整合到语言特征中,从而增强捕获目标对象的能力。此外,引入了一个语言引导的特征聚合(LGFA)模块,将语言信息整合到多尺度视觉特征中,并结合注意力缺陷补偿机制以增强特征聚合。最后,设计了一个互交互解码器(MID),通过级联双向交叉注意力增强跨模态特征对齐,从而实现精确的分割掩码预测。为进一步促进RRSIS研究,我们还构建了RISBench,一个包含52,472个图像-语言-标签三元组的新大规模基准数据集。在RISBench和其他两个流行数据集上的广泛基准测试表明,所提出的CroBIM优于现有的最先进(SOTA)方法。CroBIM的源代码和RISBench数据集将在https://github.com/HIT-SIRS/CroBIM公开提供。

英文摘要:

Given a natural language expression and a remote sensing image, the goal of referring remote sensing image segmentation (RRSIS) is to generate a pixel-level mask of the target object identified by the referring expression. In contrast to natural scenarios, expressions in RRSIS often involve complex geospatial relationships, with target objects of interest that vary significantly in scale and lack visual saliency, thereby increasing the difficulty of achieving precise segmentation. To address the aforementioned challenges, a novel RRSIS framework is proposed, termed the cross-modal bidirectional interaction model (CroBIM). Specifically, a context-aware prompt modulation (CAPM) module is designed to integrate spatial positional relationships and task-specific knowledge into the linguistic features, thereby enhancing the ability to capture the target object. Additionally, a language-guided feature aggregation (LGFA) module is introduced to integrate linguistic information into multi-scale visual features, incorporating an attention deficit compensation mechanism to enhance feature aggregation. Finally, a mutual-interaction decoder (MID) is designed to enhance cross-modal feature alignment through cascaded bidirectional cross-attention, thereby enabling precise segmentation mask prediction. To further forster the research of RRSIS, we also construct RISBench, a new large-scale benchmark dataset comprising 52,472 image-language-label triplets. Extensive benchmarking on RISBench and two other prevalent datasets demonstrates the superior performance of the proposed CroBIM over existing state-of-the-art (SOTA) methods. The source code for CroBIM and the RISBench dataset will be publicly available at https://github.com/HIT-SIRS/CroBIM

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑