arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2104.03135cs.CV
收藏

从tHe看到bOx:视觉语言表示学习的端到端预训练

Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning

University of Science and Technology Beijing(北京科技大学);Beijing Engineering Research Center of Industrial Spectrum Imaging(北京工业光谱成像工程技术研究中心);Sun Yat-sen University(中山大学);Microsoft Research Asia(微软亚洲研究院)

Zhicheng Huang, Zhaoyang Zeng, Yupan Huang, Bei Liu, Dongmei Fu, Jianlong Fu

摘要:

我们研究了卷积神经网络(CNN)和Transformer的联合学习,用于视觉语言预训练(VLPT),旨在从数百万个图像-文本对中学习跨模态对齐。最先进的方法提取显着的图像区域和对齐区域的话一步一步。由于基于区域的视觉特征通常表示图像的一部分,因此现有的视觉语言模型很难完全理解成对自然语言的语义。在本文中,我们提出SOHO“看到了tHe bOx”,需要一个完整的图像作为输入,并学习视觉语言表示在一个端到端的方式。SOHO不需要限界框注释,这使得推理速度比基于区域的方法快10倍。特别是,SOHO通过视觉词典(VD)学习提取全面而紧凑的图像特征,以促进跨模态理解。VD被设计为表示相似语义的一致视觉抽象。它是动态更新的,并在我们提出的预训练任务Masked Visual Modeling(MVM)中使用。我们进行实验,四个完善的视觉语言任务,以下标准VLPT设置。特别地,SOHO在MSCOCO文本检索5 k测试分割上实现了2.0%R@1得分的绝对增益,在NLVR $^2$ test-P分割上实现了1.5%的准确率,在SNLI-VE测试分割上实现了6.7%的准确率。

英文摘要:

We study joint learning of Convolutional Neural Network (CNN) and Transformer for vision-language pre-training (VLPT) which aims to learn cross-modal alignments from millions of image-text pairs. State-of-the-art approaches extract salient image regions and align regions with words step-by-step. As region-based visual features usually represent parts of an image, it is challenging for existing vision-language models to fully understand the semantics from paired natural languages. In this paper, we propose SOHO to "See Out of tHe bOx" that takes a whole image as input, and learns vision-language representation in an end-to-end manner. SOHO does not require bounding box annotations which enables inference 10 times faster than region-based approaches. In particular, SOHO learns to extract comprehensive yet compact image features through a visual dictionary (VD) that facilitates cross-modal understanding. VD is designed to represent consistent visual abstractions of similar semantics. It is updated on-the-fly and utilized in our proposed pre-training task Masked Visual Modeling (MVM). We conduct experiments on four well-established vision-language tasks by following standard VLPT settings. In particular, SOHO achieves absolute gains of 2.0% R@1 score on MSCOCO text retrieval 5k test split, 1.5% accuracy on NLVR$^2$ test-P split, 6.7% accuracy on SNLI-VE test split, respectively.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑