一种基于Transformer解码器的高效多任务视觉定位框架
An Efficient and Effective Transformer Decoder-Based Framework for Multi-Task Visual Grounding
AI总结：
本文提出EEVG框架，用Transformer解码器实现线性成本的语言-视觉融合，并通过无参数背景token消除和轻量掩码头高效完成多任务视觉定位。
摘要：
大多数先进的视觉定位方法依赖Transformer进行视觉-语言特征融合。然而，这些基于Transformer的方法存在一个显著缺陷：由于Transformer编码器中的自注意力机制，计算成本呈二次方增长，尤其是在处理高分辨率图像或长上下文句子时。这种计算负担的二次方增长限制了视觉定位在更复杂场景中的应用，例如涉及冗长语言表达的基于对话的推理分割。在本文中，我们提出了一种基于Transformer解码器的高效且有效的多任务视觉定位（EEVG）框架来解决这一问题，该框架在语言和视觉两个方面都降低了成本。在语言方面，我们采用Transformer解码器融合视觉和语言特征，其中语言特征作为记忆输入，视觉特征作为查询输入。这使得融合成本随语言表达长度线性扩展。在视觉方面，我们引入了一种无参数方法，通过基于注意力分数消除背景视觉token来减少计算量。然后，我们设计了一个轻量级掩码头，直接从剩余的稀疏特征图中预测分割掩码。在基准数据集上的大量结果和消融研究证明了我们方法的效率和有效性。代码可在https://github.com/chenwei746/EEVG获取。
英文摘要：
Most advanced visual grounding methods rely on Transformers for visual-linguistic feature fusion. However, these Transformer-based approaches encounter a significant drawback: the computational costs escalate quadratically due to the self-attention mechanism in the Transformer Encoder, particularly when dealing with high-resolution images or long context sentences. This quadratic increase in computational burden restricts the applicability of visual grounding to more intricate scenes, such as conversation-based reasoning segmentation, which involves lengthy language expressions. In this paper, we propose an efficient and effective multi-task visual grounding (EEVG) framework based on Transformer Decoder to address this issue, which reduces the cost in both language and visual aspects. In the language aspect, we employ the Transformer Decoder to fuse visual and linguistic features, where linguistic features are input as memory and visual features as queries. This allows fusion to scale linearly with language expression length. In the visual aspect, we introduce a parameter-free approach to reduce computation by eliminating background visual tokens based on attention scores. We then design a light mask head to directly predict segmentation masks from the remaining sparse feature maps. Extensive results and ablation studies on benchmarks demonstrate the efficiency and effectiveness of our approach. Code is available in https://github.com/chenwei746/EEVG.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
