arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2308.09779cs.CV
收藏

EAVL:显式对齐视觉与语言以用于指代图像分割

EAVL: Explicitly Align Vision and Language for Referring Image Segmentation

Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Science(中国科学院大学人工智能学院)

Yichen Yan, Xingjian He, Wenxuan Wang, Sihan Chen, Jing Liu

AI总结:

本文提出EAVL方法,通过引入Vision-Language Aligner在分割阶段使用动态卷积核显式对齐视觉与语言特征,在多个数据集上大幅超越现有方法,并可作为通用即插即用模块提升其他RIS模型性能。

摘要:

指代图像分割(RIS)旨在根据自然语言描述从图像中分割出特定对象。其主要挑战在于文本到像素的细粒度关联。在以往的方法中,最终结果通过固定卷积核获得,这遵循了与传统图像分割类似的模式。这些方法在分割阶段缺乏语言和视觉特征的显式对齐,导致关联效果次优。在本文中,我们提出了EAVL,一种显式对齐视觉和语言特征的方法。与固定卷积核不同,我们引入了Vision-Language Aligner,它基于输入图像和句子,在分割阶段使用动态卷积核对齐特征。具体而言,我们生成了多个代表语言表达不同侧重点的查询。这些查询被转换为一系列基于查询的卷积核,并在分割阶段应用以产生一系列掩码。最终结果通过聚合所有掩码获得。我们的方法在分割阶段利用了多模态特征的潜力,并将不同侧重点的语言特征与图像特征对齐,以实现细粒度的文本到像素关联。我们在RefCOCO、RefCOCO+和G-Ref上以大幅优势超越了以往的最先进方法。此外,我们的方法被设计为RIS任务中用于跨模态对齐的通用即插即用模块,易于与其他RIS模型集成以实现显著的性能提升。

英文摘要:

Referring image segmentation (RIS) aims to segment an object mentioned in natural language from an image. The main challenge is text-to-pixel fine-grained correlation. In the previous methods, the final results are obtained by convolutions with a fixed kernel, which follows a similar pattern as traditional image segmentation. These methods lack explicit alignment of language and vision features in the segmentation stage, resulting in suboptimal correlation. In this paper, we introduce EAVL, a method explicitly aligning vision and language features. In contrast to fixed convolution kernels, we introduce a Vision-Language Aligner that aligns features in the segmentation stage using dynamic convolution kernels based on the input image and sentence. Specifically, we generate multiple queries representing different emphases of language expression. These queries are transformed into a series of query-based convolution kernels, which are applied in the segmentation stage to produce a series of masks. The final result is obtained by aggregating all masks. Our method harnesses the potential of the multi-modal features in the segmentation stage and aligns language features of different emphases with image features to achieve fine-grained text-to-pixel correlation. We surpass previous state-of-the-art methods on RefCOCO, RefCOCO+, and G-Ref by large margins. Additionally, our method is designed to be a generic plug-and-play module for cross-modality alignment in RIS task, making it easy to integrate with other RIS models for substantial performance improvements.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑