基于多模态条件适配的视觉定位
Visual Grounding with Multi-modal Conditional Adaptation
AI总结：
针对视觉定位中视觉特征无法随文本描述自适应变化的问题，本文提出MMCA，利用多模态嵌入生成权重系数以动态更新视觉编码器，并在四个数据集上取得最先进性能。
摘要：
视觉定位是定位由自然语言表达所指定对象的任务。现有方法扩展通用目标检测框架来处理该任务。它们通常使用独立的视觉和文本编码器分别提取视觉与文本特征，然后在多模态解码器中融合这些特征以进行最终预测。然而，视觉定位面临独特挑战。它经常涉及在同一图像中定位具有不同文本描述的对象。现有方法在该任务上表现吃力，因为独立视觉编码器会对同一图像生成相同的视觉特征，从而限制检测性能。近期一些方法提出了各种语言引导的视觉编码器来解决这一问题，但它们大多仅依赖文本信息，并且需要复杂设计。本文提出多模态条件适配（Multi-modal Conditional Adaptation，MMCA），使视觉编码器能够自适应更新权重，将注意力引向与文本相关的区域。具体而言，我们首先整合来自不同模态的信息以获得多模态嵌入。随后，我们利用由多模态嵌入生成的一组权重系数重组权重更新矩阵，并将其应用于视觉定位模型的视觉编码器。在四个广泛使用的数据集上进行的大量实验表明，MMCA取得了显著提升和最先进结果。消融实验进一步证明了本方法的轻量性和高效性。源代码可在 https://github.com/Mr-Bigworth/MMCA 获取。
英文摘要：
Visual grounding is the task of locating objects specified by natural language expressions. Existing methods extend generic object detection frameworks to tackle this task. They typically extract visual and textual features separately using independent visual and textual encoders, then fuse these features in a multi-modal decoder for final prediction. However, visual grounding presents unique challenges. It often involves locating objects with different text descriptions within the same image. Existing methods struggle with this task because the independent visual encoder produces identical visual features for the same image, limiting detection performance. Some recently approaches propose various language-guided visual encoders to address this issue, but they mostly rely solely on textual information and require sophisticated designs. In this paper, we introduce Multi-modal Conditional Adaptation (MMCA), which enables the visual encoder to adaptively update weights, directing its focus towards text-relevant regions. Specifically, we first integrate information from different modalities to obtain multi-modal embeddings. Then we utilize a set of weighting coefficients, which generated from the multimodal embeddings, to reorganize the weight update matrices and apply them to the visual encoder of the visual grounding model. Extensive experiments on four widely used datasets demonstrate that MMCA achieves significant improvements and state-of-the-art results. Ablation experiments further demonstrate the lightweight and efficiency of our method. Our source code is available at: https://github.com/Mr-Bigworth/MMCA.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
