RC3：正则化对比跨语言跨模态预训练
RC3: Regularized Contrastive Cross-lingual Cross-modal Pre-training
AI总结：
针对多语言视觉-语言预训练依赖严格对齐数据且成本高昂的问题，提出RC^3方法，利用弱对齐数据并通过正则化对比学习约束表示，在6种语言的5个下游任务上取得更优性能。
摘要：
多语言视觉-语言（V&L）预训练在学习跨不同模态和语言的通用表示方面取得了显著进展。尽管最近取得了成功，但仍然存在一些挑战，限制了多语言环境下V&L预训练模型的进一步改进。特别是，当前的V&L预训练方法严重依赖于通过机器翻译从以英语为中心的数据集生成的严格对齐的多语言图像-文本对。然而，收集和翻译此类严格对齐的数据集的成本通常难以承受。在本文中，我们提出了正则化对比跨语言跨模态（RC^3）预训练方法，该方法进一步利用了更丰富的弱对齐多语言图像-文本对。具体来说，我们设计了一个正则化的跨语言视觉-文本对比学习目标，该目标根据文本相关性约束弱对齐的视觉-文本输入的表示接近度。此外，现有的V&L预训练方法主要通过区域兴趣（ROI）特征或补丁嵌入来处理视觉输入。我们灵活地将这两种视觉特征形式集成到我们的模型中，用于预训练和下游多模态任务。在6种语言的5个下游多模态任务上进行的大量实验证明了我们提出的方法相对于竞争对比模型的有效性，并具有更强的零样本能力。
英文摘要：
Multilingual vision-language (V&L) pre-training has achieved remarkable progress in learning universal representations across different modalities and languages. In spite of recent success, there still remain challenges limiting further improvements of V&L pre-trained models in multilingual settings. Particularly, current V&L pre-training methods rely heavily on strictly-aligned multilingual image-text pairs generated from English-centric datasets through machine translation. However, the cost of collecting and translating such strictly-aligned datasets is usually unbearable. In this paper, we propose Regularized Contrastive Cross-lingual Cross-modal (RC^3) pre-training, which further exploits more abundant weakly-aligned multilingual image-text pairs. Specifically, we design a regularized cross-lingual visio-textual contrastive learning objective that constrains the representation proximity of weakly-aligned visio-textual inputs according to textual relevance. Besides, existing V&L pre-training approaches mainly deal with visual inputs by either region-of-interest (ROI) features or patch embeddings. We flexibly integrate the two forms of visual features into our model for pre-training and downstream multi-modal tasks. Extensive experiments on 5 downstream multi-modal tasks across 6 languages demonstrate the effectiveness of our proposed method over competitive contrast models with stronger zero-shot capability.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
