用于指代图像分割的掩码定位
Mask Grounding for Referring Image Segmentation
AI总结：
针对指代图像分割中语言-图像模态差距和视觉定位监督不足的问题，提出Mask Grounding辅助任务及跨模态对齐模块，构建MagNet架构，在RefCOCO等基准上显著优于现有方法。
摘要：
指代图像分割（Referring Image Segmentation, RIS）是一项具有挑战性的任务，要求算法分割由自由形式语言表达所指代的对象。尽管近年来取得了显著进展，但大多数最先进（SOTA）的方法在像素和单词级别上仍然存在相当大的语言-图像模态差距。这些方法通常1）依赖句子级语言特征进行语言-图像对齐，以及2）缺乏用于细粒度视觉定位的显式训练监督。因此，它们在视觉和语言特征之间表现出较弱的对象级对应关系。由于缺乏良好定位的特征，先前的方法难以理解需要对多个对象之间的关系进行强推理的复杂表达，特别是在处理很少使用或模棱两可的从句时。为了应对这一挑战，我们引入了一种新颖的Mask Grounding辅助任务，通过显式教导模型学习掩码文本标记与其匹配视觉对象之间的细粒度对应关系，显著改善了语言特征内的视觉定位。Mask Grounding可以直接用于先前的RIS方法，并持续带来改进。此外，为了全面解决模态差距，我们还设计了一个跨模态对齐损失和一个配套的对齐模块。这些添加与Mask Grounding协同工作。凭借所有这些技术，我们的综合方法最终形成了MagNet（Mask-grounded Network），该架构在三个关键基准（RefCOCO、RefCOCO+和G-Ref）上显著优于先前的工作，证明了我们的方法在解决RIS算法当前局限性方面的有效性。我们的代码和预训练权重将会发布。
英文摘要：
Referring Image Segmentation (RIS) is a challenging task that requires an algorithm to segment objects referred by free-form language expressions. Despite significant progress in recent years, most state-of-the-art (SOTA) methods still suffer from considerable language-image modality gap at the pixel and word level. These methods generally 1) rely on sentence-level language features for language-image alignment and 2) lack explicit training supervision for fine-grained visual grounding. Consequently, they exhibit weak object-level correspondence between visual and language features. Without well-grounded features, prior methods struggle to understand complex expressions that require strong reasoning over relationships among multiple objects, especially when dealing with rarely used or ambiguous clauses. To tackle this challenge, we introduce a novel Mask Grounding auxiliary task that significantly improves visual grounding within language features, by explicitly teaching the model to learn fine-grained correspondence between masked textual tokens and their matching visual objects. Mask Grounding can be directly used on prior RIS methods and consistently bring improvements. Furthermore, to holistically address the modality gap, we also design a cross-modal alignment loss and an accompanying alignment module. These additions work synergistically with Mask Grounding. With all these techniques, our comprehensive approach culminates in MagNet (Mask-grounded Network), an architecture that significantly outperforms prior arts on three key benchmarks (RefCOCO, RefCOCO+ and G-Ref), demonstrating our method's effectiveness in addressing current limitations of RIS algorithms. Our code and pre-trained weights will be released.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
