arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2404.08281cs.CVcs.MM
收藏

校准与重建:用于指代图像分割的深度集成语言模型

Calibration & Reconstruction: Deep Integrated Language for Referring Image Segmentation

Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)

Yichen Yan, Xingjian He, Sihan Chen, Jing Liu

AI总结:

针对指代图像分割中传统Transformer解码器易丢失扭曲语言信息的问题,提出CRFormer模型,通过校准解码器迭代校准多模态特征,结合语言重建模块与损失防止信息损耗,在多个基准数据集上取得优于现有方法的性能。

摘要:

指代图像分割旨在从图像中分割出自然语言表述所指代的目标。该任务的核心挑战在于将细粒度语义信息从文本特征高效传递到视觉特征。近年许多研究采用Transformer应对这一挑战,但传统Transformer解码器随层数加深会扭曲语言信息,导致结果欠佳。本文提出CRFormer模型,可在Transformer解码器中迭代校准多模态特征。我们首先利用视觉特征生成语言查询,突出输入语言的不同维度。随后提出新型校准解码器(CDec),其中多模态特征可通过输入语言特征迭代校准。在校准解码器中,我们利用每一层解码器的输出与原始语言特征生成新查询以持续校准,逐步更新语言特征。基于CDec,我们引入语言重建模块与重建损失,该模块利用解码器最后一层的查询重建输入语言并计算重建损失,可进一步防止语言信息丢失或扭曲。在RefCOCO、RefCOCO+和G-Ref数据集上的实验一致表明,与现有最优方法相比,本文方法性能更优。

英文摘要:

Referring image segmentation aims to segment an object referred to by natural language expression from an image. The primary challenge lies in the efficient propagation of fine-grained semantic information from textual features to visual features. Many recent works utilize a Transformer to address this challenge. However, conventional transformer decoders can distort linguistic information with deeper layers, leading to suboptimal results. In this paper, we introduce CRFormer, a model that iteratively calibrates multi-modal features in the transformer decoder. We start by generating language queries using vision features, emphasizing different aspects of the input language. Then, we propose a novel Calibration Decoder (CDec) wherein the multi-modal features can iteratively calibrated by the input language features. In the Calibration Decoder, we use the output of each decoder layer and the original language features to generate new queries for continuous calibration, which gradually updates the language features. Based on CDec, we introduce a Language Reconstruction Module and a reconstruction loss. This module leverages queries from the final layer of the decoder to reconstruct the input language and compute the reconstruction loss. This can further prevent the language information from being lost or distorted. Our experiments consistently show the superior performance of our approach across RefCOCO, RefCOCO+, and G-Ref datasets compared to state-of-the-art methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑