SwimVG：用于视觉定位的逐步多模态融合与适应
SwimVG: Step-wise Multimodal Fusion and Adaption for Visual Grounding
AI总结：
针对视觉定位中跨模态融合不足和计算成本高的问题，提出SwimVG框架，通过逐步多模态提示和跨模态交互适配器实现参数高效的多模态融合，在四个基准上取得显著效果。
摘要：
视觉定位旨在通过自然语言定位图像区域，这严重依赖于跨模态对齐。大多数现有方法通过完全微调单模态预训练模型来分别迁移视觉/语言知识，随后简单地堆叠视觉-语言变换器进行多模态融合。然而，这些方法不仅限制了视觉和语言上下文之间的充分交互，而且产生了显著的计算成本。因此，为了解决这些问题，我们探索了一种逐步多模态融合与适应框架，即SwimVG。具体而言，SwimVG提出了逐步多模态提示（Swip）和跨模态交互适配器（CIA）用于视觉定位，取代了用于多模态融合的繁琐变换器堆叠。Swip能够以标记级融合方式逐步改善视觉和语言表示之间的对齐。此外，权重级CIA通过跨模态交互进一步促进多模态融合。Swip和CIA都是参数高效范式，它们从浅层到深层逐步融合跨模态特征。在四个广泛使用的基准上的实验结果表明，SwimVG在效率方面实现了显著的能力和可观的收益。我们的代码可在https://github.com/liuting20/SwimVG获取。
英文摘要：
Visual grounding aims to ground an image region through natural language, which heavily relies on cross-modal alignment. Most existing methods transfer visual/linguistic knowledge separately by fully fine-tuning uni-modal pre-trained models, followed by a simple stack of visual-language transformers for multimodal fusion. However, these approaches not only limit adequate interaction between visual and linguistic contexts, but also incur significant computational costs. Therefore, to address these issues, we explore a step-wise multimodal fusion and adaption framework, namely SwimVG. Specifically, SwimVG proposes step-wise multimodal prompts (Swip) and cross-modal interactive adapters (CIA) for visual grounding, replacing the cumbersome transformer stacks for multimodal fusion. Swip can improve {the} alignment between the vision and language representations step by step, in a token-level fusion manner. In addition, weight-level CIA further promotes multimodal fusion by cross-modal interaction. Swip and CIA are both parameter-efficient paradigms, and they fuse the cross-modal features from shallow to deep layers gradually. Experimental results on four widely-used benchmarks demonstrate that SwimVG achieves remarkable abilities and considerable benefits in terms of efficiency. Our code is available at https://github.com/liuting20/SwimVG.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
