arXiv 2307.123922023-07-23cs.CV

基于掩码参考中心点监督的迭代鲁棒视觉定位

Iterative Robust Visual Grounding with Masked Reference based Centerpoint Supervision

Beihang University（北京航空航天大学）；Nanyang Technological University（南洋理工大学）；University of Liverpool（利物浦大学）

Menghao Li, Chunlei Wang, Wenquan Feng, Shuchang Lyu, Guangliang Cheng, Xiangtai Li, Binghao Liu, Qi Zhao

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对现有视觉定位方法易受不准确描述干扰生成误报、特征捕捉与定位精度不足的问题，提出带MRCS的IR-VG框架，通过多级融合、点监督和误报敏感解码器提升性能，在多类数据集上达SOTA。

摘要：

视觉定位（Visual Grounding, VG）旨在根据给定的表达式从图像中定位目标对象，随着检测技术和视觉Transformer的发展，该领域已取得显著进展。然而，现有VG方法在遇到实际应用中常见的不准确或不相关描述时，往往会生成误报目标。此外，现有方法无法从整幅图像和文本描述中捕捉细粒度特征、实现精准定位以及充分理解上下文。为解决这两个问题，我们提出了一种带有掩码参考中心点监督（Masked Reference based Centerpoint Supervision, MRCS）的迭代鲁棒视觉定位（Iterative Robust Visual Grounding, IR-VG）框架。该框架引入迭代多级视觉-语言融合（Iterative Multi-level Vision-language Fusion, IMVF）以实现更好的对齐。我们利用MRCS通过逐点特征监督实现更精准的定位。随后，为提升VG的鲁棒性，我们还提出了一种多阶段误报敏感解码器（Multi-stage False-alarm Sensitive Decoder, MFSD），以防止在遇到不准确表达式时生成误报目标。我们在5个常规VG数据集和2个新构建的鲁棒VG数据集上对所提框架进行了评估。大量实验表明，IR-VG取得了新的state-of-the-art（SOTA，当前最优）结果，在两个新提出的鲁棒VG数据集上，相比现有SOTA方法分别提升了25%和10%。此外，该框架在5个常规VG数据集上也被验证有效。代码和模型将公开发布于https://github.com/cv516Buaa/IR-VG。

英文摘要：

Visual Grounding (VG) aims at localizing target objects from an image based on given expressions and has made significant progress with the development of detection and vision transformer. However, existing VG methods tend to generate false-alarm objects when presented with inaccurate or irrelevant descriptions, which commonly occur in practical applications. Moreover, existing methods fail to capture fine-grained features, accurate localization, and sufficient context comprehension from the whole image and textual descriptions. To address both issues, we propose an Iterative Robust Visual Grounding (IR-VG) framework with Masked Reference based Centerpoint Supervision (MRCS). The framework introduces iterative multi-level vision-language fusion (IMVF) for better alignment. We use MRCS to ahieve more accurate localization with point-wised feature supervision. Then, to improve the robustness of VG, we also present a multi-stage false-alarm sensitive decoder (MFSD) to prevent the generation of false-alarm objects when presented with inaccurate expressions. The proposed framework is evaluated on five regular VG datasets and two newly constructed robust VG datasets. Extensive experiments demonstrate that IR-VG achieves new state-of-the-art (SOTA) results, with improvements of 25\% and 10\% compared to existing SOTA approaches on the two newly proposed robust VG datasets. Moreover, the proposed framework is also verified effective on five regular VG datasets. Codes and models will be publicly at https://github.com/cv516Buaa/IR-VG.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。