面向视觉-语言预训练的全局与局部语义补全学习
Global and Local Semantic Completion Learning for Vision-Language Pre-training
AI总结：
针对VLP中掩码建模忽视全局语义的问题，本文提出GLSCL任务，通过MGSC与MLTC同时增强全局-局部及局部-局部跨模态对齐，并在多类视觉-语言基准上达到SOTA。
摘要：
跨模态对齐在视觉-语言预训练（VLP）模型中起着至关重要的作用，使其能够捕捉不同模态之间有意义的关联。为此，人们为VLP提出了大量掩码建模任务，以进一步促进跨模态交互。以往掩码建模任务的核心思想是专注于基于可见上下文重建被掩码的词元，从而学习局部-局部对齐。然而，它们大多很少关注为掩码数据生成的全局语义特征，导致全局表示与另一模态局部特征之间的跨模态对齐能力受限。因此，本文提出一种新颖的全局与局部语义补全学习（Global and Local Semantic Completion Learning，GLSCL）任务，以同时促进全局-局部对齐和局部-局部对齐。具体而言，GLSCL任务通过跨模态交互补全掩码数据缺失的语义，并恢复全局和局部特征。我们的GLSCL由掩码全局语义补全（MGSC）和掩码局部词元补全（MLTC）组成。MGSC促进学习更具代表性的全局特征，而这些特征对下游任务性能有很大影响；MLTC则重建模态融合的局部词元，进一步增强对多模态数据的准确理解。为评估所提方法在跨模态对齐上的表现，我们构建了一个名为ALIGN-BENCH的验证基准。此外，我们提出一种灵活的视觉编码器，使模型能够同时执行图像-文本和视频-文本多模态任务。实验结果表明，所提方法在视觉问答、图像-文本检索和视频-文本检索等多种视觉-语言基准上取得了最先进性能。
英文摘要：
Cross-modal alignment plays a crucial role in vision-language pre-training (VLP) models, enabling them to capture meaningful associations across different modalities. For this purpose, numerous masked modeling tasks have been proposed for VLP to further promote cross-modal interactions. The core idea of previous masked modeling tasks is to focus on reconstructing the masked tokens based on visible context for learning local-local alignment. However, most of them pay little attention to the global semantic features generated for the masked data, resulting in a limited cross-modal alignment ability of global representations to local features of the other modality. Therefore, in this paper, we propose a novel Global and Local Semantic Completion Learning (GLSCL) task to facilitate global-local alignment and local-local alignment simultaneously. Specifically, the GLSCL task complements the missing semantics of masked data and recovers global and local features by cross-modal interactions. Our GLSCL consists of masked global semantic completion (MGSC) and masked local token completion (MLTC). MGSC promotes learning more representative global features, which have a great impact on the performance of downstream tasks, while MLTC reconstructs modal-fusion local tokens, further enhancing accurate comprehension of multimodal data. To evaluate the proposed approaches on cross-modal alignment, we develop a validation benchmark called ALIGN-BENCH. Moreover, we present a flexible vision encoder, enabling our model to simultaneously perform image-text and video-text multimodal tasks. Experimental results show that our proposed method obtains state-of-the-art performance on various vision-language benchmarks, such as visual question answering, image-text retrieval, and video-text retrieval.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
