基于粗到细一致性约束的多任务视觉定位
Multi-task Visual Grounding with Coarse-to-Fine Consistency Constraints
AI总结：
针对多任务视觉定位中REC与RIS预测不一致、多模态理解不足的问题，提出两阶段C³VG架构，通过粗感知加一致性交互优化及预训练模型，在多个数据集上性能显著超越现有最优方法。
摘要：
多任务视觉定位指的是根据文本描述同时在图像中执行定位与分割任务。大多数先进方法主要聚焦于基于transformer的多模态融合，旨在提取鲁棒的多模态表征。然而，指代表达理解（REC）与指代图像分割（RIS）之间的歧义性容易引发错误，导致多任务预测结果不一致。此外，多模态理解不足会直接造成目标感知偏差。为解决这些挑战，我们提出了一种粗到细一致性约束视觉定位架构（$\text{C}^3\text{VG}$），该架构在两阶段框架内整合了隐式与显式建模方法。首先，采用查询解码器与像素解码器生成初步的检测与分割输出，这一阶段被称为粗略语义感知（RSP）阶段。随后，这些粗略预测结果通过所提出的掩码引导交互模块（MIM）以及一种新型显式双向一致性约束损失进行优化，以确保跨任务的表征一致性，我们将这一阶段称为精细化一致性交互（RCI）阶段。此外，为应对多模态理解不足的挑战，我们利用了基于视觉-语言融合表征的预训练模型。在RefCOCO、RefCOCO+和RefCOCOg数据集上的实验评估证明了$\text{C}^3\text{VG}$的有效性与合理性，其性能大幅优于当前最优的REC和RIS方法。代码与模型将发布于https://github.com/Dmmm1997/C3VG。
英文摘要：
Multi-task visual grounding involves the simultaneous execution of localization and segmentation in images based on textual expressions. The majority of advanced methods predominantly focus on transformer-based multimodal fusion, aiming to extract robust multimodal representations. However, ambiguity between referring expression comprehension (REC) and referring image segmentation (RIS) is error-prone, leading to inconsistencies between multi-task predictions. Besides, insufficient multimodal understanding directly contributes to biased target perception. To overcome these challenges, we propose a Coarse-to-fine Consistency Constraints Visual Grounding architecture ($\text{C}^3\text{VG}$), which integrates implicit and explicit modeling approaches within a two-stage framework. Initially, query and pixel decoders are employed to generate preliminary detection and segmentation outputs, a process referred to as the Rough Semantic Perception (RSP) stage. These coarse predictions are subsequently refined through the proposed Mask-guided Interaction Module (MIM) and a novel explicit bidirectional consistency constraint loss to ensure consistent representations across tasks, which we term the Refined Consistency Interaction (RCI) stage. Furthermore, to address the challenge of insufficient multimodal understanding, we leverage pre-trained models based on visual-linguistic fusion representations. Empirical evaluations on the RefCOCO, RefCOCO+, and RefCOCOg datasets demonstrate the efficacy and soundness of $\text{C}^3\text{VG}$, which significantly outperforms state-of-the-art REC and RIS methods by a substantial margin. Code and model will be available at \url{https://github.com/Dmmm1997/C3VG}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
