增强配对：面向基于定位的视觉语言模型的语义保持型图文配对增强
Augment the Pairs: Semantics-Preserving Image-Caption Pair Augmentation for Grounding-Based Vision and Language Models
AI总结：
针对基于定位的视觉语言模型训练数据规模受限、图文配对增强难度大的问题，提出结合文本条件增强与像素级掩码的语义保持型增强方法，在多个数据集上取得SOTA性能。
摘要：
基于定位的视觉语言模型已成功应用于低层视觉任务，旨在精准定位字幕中提及的物体。定位表示学习的有效性很大程度上依赖于训练数据集的规模。数据增强虽是一种有效的数据丰富策略，但在现有视觉语言任务中受到的关注极少，因为图文配对的增强并非易事。本研究中，我们提出一种鲁棒的短语定位模型，采用文本条件与非文本条件的数据增强方式训练。具体而言，我们应用文本条件的颜色抖动和水平翻转，确保图像与字幕之间的语义一致性。为保证训练样本中的图文对应关系，我们在应用水平翻转时根据预定义关键词修改字幕。此外，受近期掩码信号重建研究的启发，我们提出将像素级掩码作为一种新型数据增强形式。我们在MDETR框架上验证了该数据增强方法，同时所提方法也适用于采用其他框架的常见基于定位的视觉语言任务。最后，我们表明在大规模图文数据集（如CLIP）上预训练的图像编码器可进一步提升效果。通过在Flickr30k、指代表达和GQA三个常用数据集上的大量实验，我们的方法在多项指标上均展现出优于现有最优水平的性能。代码可在https://github.com/amzn/augment-the-pairs-wacv2024获取。
英文摘要：
Grounding-based vision and language models have been successfully applied to low-level vision tasks, aiming to precisely locate objects referred in captions. The effectiveness of grounding representation learning heavily relies on the scale of the training dataset. Despite being a useful data enrichment strategy, data augmentation has received minimal attention in existing vision and language tasks as augmentation for image-caption pairs is non-trivial. In this study, we propose a robust phrase grounding model trained with text-conditioned and text-unconditioned data augmentations. Specifically, we apply text-conditioned color jittering and horizontal flipping to ensure semantic consistency between images and captions. To guarantee image-caption correspondence in the training samples, we modify the captions according to pre-defined keywords when applying horizontal flipping. Additionally, inspired by recent masked signal reconstruction, we propose to use pixel-level masking as a novel form of data augmentation. While we demonstrate our data augmentation method with MDETR framework, the proposed approach is applicable to common grounding-based vision and language tasks with other frameworks. Finally, we show that image encoder pretrained on large-scale image and language datasets (such as CLIP) can further improve the results. Through extensive experiments on three commonly applied datasets: Flickr30k, referring expressions and GQA, our method demonstrates advanced performance over the state-of-the-arts with various metrics. Code can be found in https://github.com/amzn/augment-the-pairs-wacv2024.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
