arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.07539cs.CVcs.AI
收藏

面向指代图像分割的带阶段划分视觉与语言Transformer编码器的交叉感知早期融合

Cross-aware Early Fusion with Stage-divided Vision and Language Transformer Encoders for Referring Image Segmentation

School of Artificial Intelligence, Sogang University(西江大学人工智能学院);School of Electronic Engineering, Sogang University(西江大学电子工程学院)

Yubin Cho, Hyunwoo Yu, Suk-ju Kang

AI总结:

针对指代图像分割中早期融合时语言特征无法参照视觉信息的问题,提出CrossVLT架构,通过阶段划分的视觉与语言Transformer编码器实现双向交叉感知早期融合与多层级特征对齐,在三个公开基准上性能超越现有最优方法。

摘要:

指代分割旨在分割与自然语言表述相关的目标物体。该任务的核心挑战在于理解复杂且存在歧义的语言表述的含义,并通过参照该表述在包含多个物体的图像中确定对应的相关区域。近期的模型聚焦于在视觉编码器的中间阶段与语言特征进行早期融合,但这些方法存在局限性:语言特征无法参照视觉信息。为解决这一问题,本文提出了一种新型架构——带阶段划分视觉与语言Transformer编码器的交叉感知早期融合(CrossVLT),该架构允许视觉与语言编码器共同执行早期融合,以提升跨模态上下文建模的能力。与以往方法不同,我们的方法使视觉和语言特征能够在每个阶段参照彼此的信息,相互增强两个编码器的鲁棒性。此外,不同于仅依赖高层特征进行跨模态对齐的传统方案,我们引入了一种基于特征的对齐方案,该方案使视觉与语言编码器的从低层到高层的特征都能参与跨模态对齐。通过在所有编码器阶段对齐中间跨模态特征,该方案实现了有效的跨模态融合。通过这种方式,所提出的方法对于指代图像分割而言简单且有效,并且在三个公开基准上的表现优于以往的最优方法。

英文摘要:

Referring segmentation aims to segment a target object related to a natural language expression. Key challenges of this task are understanding the meaning of complex and ambiguous language expressions and determining the relevant regions in the image with multiple objects by referring to the expression. Recent models have focused on the early fusion with the language features at the intermediate stage of the vision encoder, but these approaches have a limitation that the language features cannot refer to the visual information. To address this issue, this paper proposes a novel architecture, Cross-aware early fusion with stage-divided Vision and Language Transformer encoders (CrossVLT), which allows both language and vision encoders to perform the early fusion for improving the ability of the cross-modal context modeling. Unlike previous methods, our method enables the vision and language features to refer to each other's information at each stage to mutually enhance the robustness of both encoders. Furthermore, unlike the conventional scheme that relies solely on the high-level features for the cross-modal alignment, we introduce a feature-based alignment scheme that enables the low-level to high-level features of the vision and language encoders to engage in the cross-modal alignment. By aligning the intermediate cross-modal features in all encoder stages, this scheme leads to effective cross-modal fusion. In this way, the proposed approach is simple but effective for referring image segmentation, and it outperforms the previous state-of-the-art methods on three public benchmarks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑