改进跨语言跨模态检索中的一致性：基于1-to-K对比学习
Improving the Consistency in Cross-Lingual Cross-Modal Retrieval with 1-to-K Contrastive Learning
AI总结：
针对跨语言跨模态检索中模态内错误传播与模态间优化偏差导致的不一致问题，提出1-to-K对比学习方法和MRV评估指标，在四个数据集上以更小预训练数据提升召回率与一致性。
摘要：
跨语言跨模态检索（CCR）是网络搜索中的一项基本任务，旨在同时打破模态和语言之间的障碍，并在多语言场景下通过单一模型实现图像-文本检索。近年来，基于跨语言跨模态预训练的方法取得了显著进展，尤其是基于大规模数据对比学习的方法大幅提升了检索任务的性能。然而，这些方法直接沿用跨语言或跨模态领域中现有的预训练方法，导致CCR中存在两个不一致性问题：跨语言风格的方法遭受模态内错误传播，导致整个数据集上不同语言的召回性能不一致；跨模态风格的方法遭受模态间优化方向偏差，导致每个实例内不同语言的排名不一致，而这种不一致无法通过Recall@K反映。为解决这些问题，我们提出了一种简单但有效的1-to-K对比学习方法，该方法平等对待每种语言，消除了错误传播和优化偏差。此外，我们提出了一种新的评估指标——平均排名方差（MRV），以反映每个实例内不同语言间的排名不一致性。在四个CCR数据集上的大量实验表明，我们的方法在较小规模的预训练数据下同时提高了召回率和MRV，达到了新的最先进水平。
英文摘要：
Cross-lingual Cross-modal Retrieval (CCR) is an essential task in web search, which aims to break the barriers between modality and language simultaneously and achieves image-text retrieval in the multi-lingual scenario with a single model. In recent years, excellent progress has been made based on cross-lingual cross-modal pre-training; particularly, the methods based on contrastive learning on large-scale data have significantly improved retrieval tasks. However, these methods directly follow the existing pre-training methods in the cross-lingual or cross-modal domain, leading to two problems of inconsistency in CCR: The methods with cross-lingual style suffer from the intra-modal error propagation, resulting in inconsistent recall performance across languages in the whole dataset. The methods with cross-modal style suffer from the inter-modal optimization direction bias, resulting in inconsistent rank across languages within each instance, which cannot be reflected by Recall@K. To solve these problems, we propose a simple but effective 1-to-K contrastive learning method, which treats each language equally and eliminates error propagation and optimization bias. In addition, we propose a new evaluation metric, Mean Rank Variance (MRV), to reflect the rank inconsistency across languages within each instance. Extensive experiments on four CCR datasets show that our method improves both recall rates and MRV with smaller-scale pre-trained data, achieving the new state-of-art.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
