SimVG：一种采用解耦多模态融合的视觉定位简单框架
SimVG: A Simple Framework for Visual Grounding with Decoupled Multi-modal Fusion
AI总结：
本文提出 SimVG 框架，通过预训练模型解耦视觉-语言融合、引入物体 token 并采用动态权重平衡蒸馏，在六个 VG 基准上实现更快推理与最先进性能。
摘要：
视觉定位是一项常见的视觉任务，旨在将描述性句子定位到图像中的相应区域。大多数现有方法采用独立的图像-文本编码，并使用复杂的手工设计模块或编码器-解码器架构进行模态交互和查询推理。然而，在处理复杂文本表达时，它们的性能会显著下降。这是因为前一种范式仅利用有限的下游数据来拟合多模态特征融合，因此只在文本表达相对简单时有效。相比之下，鉴于文本表达的广泛多样性以及下游训练数据的独特性，现有从视觉-语言上下文中提取多模态内容的融合模块尚未得到充分研究。本文提出了一个简单而稳健的基于 Transformer 的视觉定位框架 SimVG。具体而言，我们利用现有的多模态预训练模型，将视觉-语言特征融合与下游任务解耦，并引入额外的物体 token，以促进下游任务与预训练任务的深度整合。此外，我们在多分支同步学习过程中设计了一种动态权重平衡蒸馏方法，以增强较简单分支的表示能力。该分支仅由轻量级 MLP 组成，从而简化了结构并提高了推理速度。在六个广泛使用的 VG 数据集（即 RefCOCO/+/g、ReferIt、Flickr30K 和 GRefCOCO）上的实验证明了 SimVG 的优越性。最后，所提出的方法不仅提升了效率和收敛速度，还在这些基准上取得了新的最先进性能。代码和模型将在 https://github.com/Dmmm1997/SimVG 提供。
英文摘要：
Visual grounding is a common vision task that involves grounding descriptive sentences to the corresponding regions of an image. Most existing methods use independent image-text encoding and apply complex hand-crafted modules or encoder-decoder architectures for modal interaction and query reasoning. However, their performance significantly drops when dealing with complex textual expressions. This is because the former paradigm only utilizes limited downstream data to fit the multi-modal feature fusion. Therefore, it is only effective when the textual expressions are relatively simple. In contrast, given the wide diversity of textual expressions and the uniqueness of downstream training data, the existing fusion module, which extracts multimodal content from a visual-linguistic context, has not been fully investigated. In this paper, we present a simple yet robust transformer-based framework, SimVG, for visual grounding. Specifically, we decouple visual-linguistic feature fusion from downstream tasks by leveraging existing multimodal pre-trained models and incorporating additional object tokens to facilitate deep integration of downstream and pre-training tasks. Furthermore, we design a dynamic weight-balance distillation method in the multi-branch synchronous learning process to enhance the representation capability of the simpler branch. This branch only consists of a lightweight MLP, which simplifies the structure and improves reasoning speed. Experiments on six widely used VG datasets, i.e., RefCOCO/+/g, ReferIt, Flickr30K, and GRefCOCO, demonstrate the superiority of SimVG. Finally, the proposed method not only achieves improvements in efficiency and convergence speed but also attains new state-of-the-art performance on these benchmarks. Codes and models will be available at \url{https://github.com/Dmmm1997/SimVG}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
