原型感知多模态对齐用于开放词汇视觉定位
Prototype-Aware Multimodal Alignment for Open-Vocabulary Visual Grounding
AI总结：
提出原型感知多模态学习框架PAML，通过ALBEF对齐、视觉判别编码器和原型发现继承机制，解决开放词汇视觉定位中的模态对齐、特征融合和原型利用问题，在五个基准上达到最先进性能。
摘要：
视觉定位（Visual Grounding, VG）旨在利用给定的自然语言查询在图像中定位特定目标对象。虽然当前基于Transformer的方法在标准场景（即没有新对象的场景）中展现出强大的定位性能，但它们在开放词汇场景（即测试时包含熟悉和新颖对象类别）中表现出明显的局限性。这些局限性主要源于三个关键因素：（1）视觉和语言模态之间的对齐不完美，（2）跨模态特征融合不足，以及（3）语义原型信息的利用不充分。为了克服这些挑战，我们提出了原型感知多模态学习（Prototype-Aware Multimodal Learning, PAML），一个创新的框架，通过几个关键组件系统地解决这些问题：首先，我们利用ALBEF在初始特征编码阶段建立稳健的跨模态对齐。随后，我们的视觉判别特征编码器（Visual Discriminative Feature Encoder）选择性地增强显著对象表示，同时抑制不相关的视觉上下文。该框架随后引入了一种新颖的原型发现与继承机制，提取并聚合多邻居语义原型以促进开放词汇识别。这些增强的特征通过我们的多阶段解码器（Multi-stage Decoder）进行全面的多模态集成，然后进行最终的边界框回归。在五个基准数据集上的大量实验验证了我们的方法，在标准场景中表现出有竞争力的性能，同时在开放词汇场景中取得了最先进的结果。我们的代码可在 https://github.com/plankXie/PAML 获取。
英文摘要：
Visual Grounding (VG) aims to utilize given natural language queries to locate specific target objects within images. While current transformer-based approaches demonstrate strong localization performance in standard scene (i.e, scenarios without any novel objects), they exhibit notable limitations in open-vocabulary scene (i.e, both familiar and novel object categories during testing). These limitations primarily stem from three key factors: (1) imperfect alignment between visual and linguistic modalities, (2) insufficient cross-modal feature fusion, and (3) ineffective utilization of semantic prototype information. To overcome these challenges, we present Prototype-Aware Multimodal Learning (PAML), an innovative framework that systematically addresses these issues through several key components: First, we leverage ALBEF to establish robust cross-modal alignment during initial feature encoding. Subsequently, our Visual Discriminative Feature Encoder selectively enhances salient object representations while suppressing irrelevant visual context. The framework then incorporates a novel prototype discovering and inheriting mechanism that extracts and aggregates multi-neighbor semantic prototypes to facilitate open-vocabulary recognition. These enriched features undergo comprehensive multimodal integration through our Multi-stage Decoder before final bounding box regression. Extensive experiments across five benchmark datasets validate our approach, showing competitive performance in standard scene while achieving state-of-the-art results in open-vocabulary scene. Our code is available at https://github.com/plankXie/PAML.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
