arXiv 2204.099572024-03-12cs.CV

面向指代表达理解的自步多粒度跨模态交互建模

Self-paced Multi-grained Cross-modal Interaction Modeling for Referring Expression Comprehension

School of Software Technology, Zhejiang University（浙江大学软件学院）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；Zhejiang University-University of Illinois at Urbana-Champaign Institute, Zhejiang University（浙江大学-伊利诺伊大学厄巴纳-香槟分校研究院）

Peihan Miao, Wei Su, Gaoang Wang, Xuewei Li, Xi Li

备注：: Accepted by TIP

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对指代表达理解中多粒度信息聚合与难例知识提取的问题，提出自步多粒度跨模态交互建模框架，结合多粒度跨模态注意力与自步样本学习，在多个基准数据集上取得SOTA性能。

摘要：

作为视觉-语言任务中一项重要且具有挑战性的问题，指代表达理解（referring expression comprehension，REC）通常需要大量视觉与语言模态的多粒度信息来实现精准推理。此外，由于视觉场景的多样性和语言表达的差异性，部分难例拥有比其他样本丰富得多的多粒度信息。如何聚合来自不同模态的多粒度信息，并从难例中提取丰富知识，是REC任务的关键。为解决上述挑战，本文提出一种自步多粒度跨模态交互建模框架，通过网络结构与学习机制的创新提升语言到视觉的定位能力。具体而言，我们设计了一种基于transformer的多粒度跨模态注意力，有效利用视觉与语言编码器中固有的多粒度信息。此外，考虑到样本差异较大，我们提出自步样本信息量学习，自适应地强化网络对包含丰富多粒度信息的样本的学习。该框架在RefCOCO、RefCOCO+、RefCOCOg和ReferItGame等广泛使用的数据集上显著优于当前最优方法，证明了方法的有效性。

英文摘要：

As an important and challenging problem in vision-language tasks, referring expression comprehension (REC) generally requires a large amount of multi-grained information of visual and linguistic modalities to realize accurate reasoning. In addition, due to the diversity of visual scenes and the variation of linguistic expressions, some hard examples have much more abundant multi-grained information than others. How to aggregate multi-grained information from different modalities and extract abundant knowledge from hard examples is crucial in the REC task. To address aforementioned challenges, in this paper, we propose a Self-paced Multi-grained Cross-modal Interaction Modeling framework, which improves the language-to-vision localization ability through innovations in network structure and learning mechanism. Concretely, we design a transformer-based multi-grained cross-modal attention, which effectively utilizes the inherent multi-grained information in visual and linguistic encoders. Furthermore, considering the large variance of samples, we propose a self-paced sample informativeness learning to adaptively enhance the network learning for samples containing abundant multi-grained information. The proposed framework significantly outperforms state-of-the-art methods on widely used datasets, such as RefCOCO, RefCOCO+, RefCOCOg, and ReferItGame datasets, demonstrating the effectiveness of our method.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。