arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2106.01804cs.CVcs.AIcs.CL
收藏

E2E-VLP:通过视觉学习增强的端到端视觉语言预训练

E2E-VLP: End-to-End Vision-Language Pre-training Enhanced by Visual Learning

Alibaba Group(阿里巴巴集团)

Haiyang Xu, Ming Yan, Chenliang Li, Bin Bi, Songfang Huang, Wenming Xiao, Fei Huang

摘要:

大规模图像-文本对的视觉语言预训练(VLP)在跨模态下游任务中取得了巨大的成功。现有的预训练方法主要采用两步训练的方法,首先利用预训练的目标检测器提取基于区域的视觉特征,然后将图像表示和文本嵌入作为Transformer的输入进行训练。然而,这些方法面临的问题,使用特定任务的特定对象检测器的通用跨模态理解的视觉表示,和两个阶段的流水线的计算效率低下。在本文中,我们提出了第一个用于V+L理解和生成的端到端视觉语言预训练模型,即E2 E-VLP,在其中我们构建了一个统一的Transformer框架,以共同学习视觉表示以及图像和文本之间的语义对齐。我们将目标检测和图像描述的任务与一个统一的Transformer编码器-解码器架构,以增强视觉学习的预训练。一组广泛的实验已经进行了良好的视觉语言下游任务,以证明这种新的VLP范式的有效性。

英文摘要:

Vision-language pre-training (VLP) on large-scale image-text pairs has achieved huge success for the cross-modal downstream tasks. The most existing pre-training methods mainly adopt a two-step training procedure, which firstly employs a pre-trained object detector to extract region-based visual features, then concatenates the image representation and text embedding as the input of Transformer to train. However, these methods face problems of using task-specific visual representation of the specific object detector for generic cross-modal understanding, and the computation inefficiency of two-stage pipeline. In this paper, we propose the first end-to-end vision-language pre-trained model for both V+L understanding and generation, namely E2E-VLP, where we build a unified Transformer framework to jointly learn visual representation, and semantic alignments between image and text. We incorporate the tasks of object detection and image captioning into pre-training with a unified Transformer encoder-decoder architecture for enhancing visual learning. An extensive set of experiments have been conducted on well-established vision-language downstream tasks to demonstrate the effectiveness of this novel VLP paradigm.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑