TiMix：面向高效视觉-语言预训练的文本感知图像混合方法
TiMix: Text-aware Image Mixing for Effective Vision-Language Pre-training
AI总结：
本文提出文本感知图像混合方法TiMix，将数据混合增强融入自监督多模态对比学习，并从互信息角度证明其正则化作用，在减少数据和训练时间的情况下仍保持可比性能。
摘要：
自监督多模态对比学习（SMCL）通过对齐视觉与语言模态，显著推动了现代视觉-语言预训练（VLP）模型的发展。然而，由于网络采集的图文对中存在噪声，在SMCL中扩大训练数据规模在计算成本和数据效率方面面临相当大的障碍。为了提高VLP中的数据效率，我们提出了文本感知图像混合（TiMix），它将基于混合的数据增强技术集成到SMCL中，在不显著增加计算开销的情况下带来显著的性能提升。我们从互信息（MI）的角度对TiMix进行了理论分析，表明用于跨模态对比学习的混合数据样本隐式地充当对比损失的正则化器。实验结果表明，与现有方法进行基准对比时，即使在减少训练数据量和缩短训练时间的情况下，TiMix在下游任务上仍表现出可比的性能。这项工作从实证和理论两方面证明了数据混合在数据高效且计算可行的VLP中的潜力，有利于VLP模型在实际场景中得到更广泛的采用。
英文摘要：
Self-supervised Multi-modal Contrastive Learning (SMCL) remarkably advances modern Vision-Language Pre-training (VLP) models by aligning visual and linguistic modalities. Due to noises in web-harvested text-image pairs, however, scaling up training data volume in SMCL presents considerable obstacles in terms of computational cost and data inefficiency. To improve data efficiency in VLP, we propose Text-aware Image Mixing (TiMix), which integrates mix-based data augmentation techniques into SMCL, yielding significant performance improvements without significantly increasing computational overhead. We provide a theoretical analysis of TiMixfrom a mutual information (MI) perspective, showing that mixed data samples for cross-modal contrastive learning implicitly serve as a regularizer for the contrastive loss. The experimental results demonstrate that TiMix exhibits a comparable performance on downstream tasks, even with a reduced amount of training data and shorter training time, when benchmarked against existing methods. This work empirically and theoretically demonstrates the potential of data mixing for data-efficient and computationally viable VLP, benefiting broader VLP model adoption in practical scenarios.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
