精细化视觉语言建模用于细粒度多模态预训练
Refined Vision-Language Modeling for Fine-grained Multi-modal Pre-training
AI总结：
本文提出一种无需对象注释的细粒度视觉语言预训练方案，通过同音词重写算法提供词元级监督，并设计精细化对比、匹配和替换语言建模任务，在下游任务上表现优越。
摘要：
基于对象注释的细粒度监督已广泛用于视觉和语言预训练（VLP）。然而，在现实世界的应用场景中，对齐的多模态数据通常采用图像-标题格式，这仅提供粗粒度的监督。为不同场景收集对象注释并构建对象注释预提取器不仅成本高昂，而且计算开销大。在本文中，我们从语言学角度提出了一种无需对象注释的细粒度VLP方案。首先，我们提出了一种同音词句子重写（HSR）算法以提供词元级监督。该算法将标题中的动词/名词/形容词/量词替换为其在WordNet中的同音词。相应地，我们提出了精细化视觉语言建模（RVLM）框架以利用词元级监督。提出了三个精细化任务，即精细化图像-文本对比（RITC）、精细化图像-文本匹配（RITM）和替换语言建模（RLM），以学习细粒度对齐。在多个下游任务上的大量实验证明了所提出方法的优越性能。
英文摘要：
Fine-grained supervision based on object annotations has been widely used for vision and language pre-training (VLP). However, in real-world application scenarios, aligned multi-modal data is usually in the image-caption format, which only provides coarse-grained supervision. It is not only cost-expensive but also compute-expensive to collect object annotations and build object annotation pre-extractor for different scenarios. In this paper, we propose a fine-grained VLP scheme without object annotations from the linguistic perspective. First, we propose a homonym sentence rewriting (HSR) algorithm to provide token-level supervision. The algorithm replaces a verb/noun/adjective/quantifier word of the caption with its homonyms from WordNet. Correspondingly, we propose refined vision-language modeling (RVLM) framework to exploit the token-level supervision. Three refined tasks, i.e., refined image-text contrastive (RITC), refined image-text matching (RITM), and replace language modeling (RLM) are proposed to learn the fine-grained alignment. Extensive experiments on several downstream tasks demonstrate the superior performance of the proposed method.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
