通过有效的跨模态蒸馏弥合视觉定位中的模态差距
Bridging Modality Gap for Visual Grounding with Effecitve Cross-modal Distillation
AI总结：
本文提出EpmVG框架，通过跨模态蒸馏机制将多模态预训练模型中的图文一致性信息引入视觉定位任务，以弥合单模态骨干网络之间的领域差距，并在五个数据集上验证了其优于现有方法的性能。
摘要：
视觉定位旨在将图像特定区域的视觉信息与相应的自然语言表达对齐。当前的视觉定位方法分别利用预训练的视觉和语言骨干网络来获取视觉特征和语言特征。尽管这两类特征随后通过精心设计的网络进行融合，但特征的异质性使其不适合多模态推理。这一问题源于当前视觉定位方法所使用的单模态预训练骨干网络之间的领域差距，传统的端到端训练方法难以弥合这一差距。为了缓解这一问题，本文提出了一个用于视觉定位的赋能预训练模型框架（EpmVG），该框架蒸馏一个多模态预训练模型来指导视觉定位任务。EpmVG依赖一种新颖的跨模态蒸馏机制，能够有效地从预训练模型中引入图像和文本的一致性信息，减少骨干网络中的领域差距，从而提升模型在视觉定位任务中的性能。在五个常用数据集上进行了大量实验，结果表明我们的方法取得了优于最先进方法的性能。
英文摘要：
Visual grounding aims to align visual information of specific regions of images with corresponding natural language expressions. Current visual grounding methods leverage pre-trained visual and language backbones independently to obtain visual features and linguistic features. Although these two types of features are then fused through elaborately designed networks, the heterogeneity of the features renders them unsuitable for multi-modal reasoning. This problem arises from the domain gap between the single-modal pre-training backbones used in current visual grounding methods, which can hardly be bridged by the traditional end-to-end training method. To alleviate this, our work proposes an Empowering Pre-trained Model for Visual Grounding (EpmVG) framework, which distills a multimodal pre-trained model to guide the visual grounding task. EpmVG relies on a novel cross-modal distillation mechanism that can effectively introduce the consistency information of images and texts from the pre-trained model, reducing the domain gap in the backbone networks, and thereby improving the performance of the model in the visual grounding task. Extensive experiments have been conducted on five conventionally used datasets, and the results demonstrate that our method achieves better performance than state-of-the-art methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
