PC$^2$：跨模态检索中面向噪声对应学习的基于伪分类的伪描述生成
PC$^2$: Pseudo-Classification Based Pseudo-Captioning for Noisy Correspondence Learning in Cross-Modal Retrieval
AI总结：
针对跨模态检索中的噪声对应学习问题，提出PC$^2$框架，通过伪分类、伪描述监督和对应修正提升鲁棒性，并发布真实噪声数据集NoW。
摘要：
在跨模态检索领域，将多媒体中的不同模态无缝整合仍是一项艰巨挑战，尤其是在噪声对应学习（NCL）带来复杂性的情况下。此类噪声通常源于不匹配的数据对，这是一种区别于传统噪声标签的重要障碍。本文提出基于伪分类的伪描述生成（Pseudo-Classification based Pseudo-Captioning，PC$^2$）框架以应对该挑战。PC$^2$提供三方面策略：首先，它建立一个辅助的“伪分类”任务，将描述解释为类别标签，引导模型通过非对比机制学习图文语义相似性。其次，不同于当前主流的基于间隔的方法，我们利用PC$^2$的伪分类能力生成伪描述，为每个不匹配数据对提供更具信息量且更切实的监督。第三，借助伪分类的振荡来辅助修正对应关系。除技术贡献外，我们构建了一个名为Noise of Web（NoW）的真实NCL数据集，它可成为噪声自然存在的新型强大NCL基准。在多种NCL设置下，对PC$^2$的实证评估表明，其在模拟数据集和真实数据集上均显著优于现有的最先进鲁棒跨模态检索方法。所贡献的数据集和源代码已发布于https://github.com/alipay/PC2-NoiseofWeb。
英文摘要：
In the realm of cross-modal retrieval, seamlessly integrating diverse modalities within multimedia remains a formidable challenge, especially given the complexities introduced by noisy correspondence learning (NCL). Such noise often stems from mismatched data pairs, which is a significant obstacle distinct from traditional noisy labels. This paper introduces Pseudo-Classification based Pseudo-Captioning (PC$^2$) framework to address this challenge. PC$^2$ offers a threefold strategy: firstly, it establishes an auxiliary "pseudo-classification" task that interprets captions as categorical labels, steering the model to learn image-text semantic similarity through a non-contrastive mechanism. Secondly, unlike prevailing margin-based techniques, capitalizing on PC$^2$'s pseudo-classification capability, we generate pseudo-captions to provide more informative and tangible supervision for each mismatched pair. Thirdly, the oscillation of pseudo-classification is borrowed to assistant the correction of correspondence. In addition to technical contributions, we develop a realistic NCL dataset called Noise of Web (NoW), which could be a new powerful NCL benchmark where noise exists naturally. Empirical evaluations of PC$^2$ showcase marked improvements over existing state-of-the-art robust cross-modal retrieval techniques on both simulated and realistic datasets with various NCL settings. The contributed dataset and source code are released at https://github.com/alipay/PC2-NoiseofWeb.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
