扩展多模态对比表示
Extending Multi-modal Contrastive Representations
AI总结：
本文提出Ex-MCR方法，通过将现有MCR空间对齐到基础空间，无需配对数据即可学习超过三种模态的统一对比表示，在多项检索和分类任务上达到最先进性能。
摘要：
超过三种模态的多模态对比表示（MCR）在多模态学习中至关重要。尽管近期方法取得了令人瞩目的成就，但对大规模、高质量配对数据的高度依赖以及昂贵的训练成本限制了其进一步发展。受近期C-MCR的启发，本文提出了扩展多模态对比表示（Ex-MCR），一种训练高效且无需配对数据的方法，通过整合现有MCR空间的知识，灵活地学习超过三种模态的统一对比表示空间。具体而言，Ex-MCR将多个现有的MCR对齐到同一个基础MCR中，这能有效保留基础MCR原有的语义对齐。此外，我们从训练数据、架构和学习目标的角度全面增强了整个对齐MCR空间的学习流程。凭借保留的原始模态对齐和增强的空间对齐，Ex-MCR展现出优越的表示学习性能和出色的模态可扩展性。为证明Ex-MCR的有效性，我们将CLAP（音频-文本）和ULIP（3D-视觉）的MCR空间分别利用重叠的文本和图像模态对齐到CLIP（视觉-文本）中。值得注意的是，在不使用任何配对数据的情况下，Ex-MCR学习了一个3D-图像-文本-音频统一的对比表示，并在音频-视觉、3D-图像、音频-文本、视觉-文本检索以及3D物体分类任务上达到了最先进的性能。更重要的是，大量的定性结果进一步展示了扩展模态（如音频和3D）之间涌现的语义对齐，这凸显了模态可扩展性的巨大潜力。
英文摘要：
Multi-modal contrastive representation (MCR) of more than three modalities is critical in multi-modal learning. Although recent methods showcase impressive achievements, the high dependence on large-scale, high-quality paired data and the expensive training costs limit their further development. Inspired by recent C-MCR, this paper proposes Extending Multimodal Contrastive Representation (Ex-MCR), a training-efficient and paired-data-free method to flexibly learn unified contrastive representation space for more than three modalities by integrating the knowledge of existing MCR spaces. Specifically, Ex-MCR aligns multiple existing MCRs into the same based MCR, which can effectively preserve the original semantic alignment of the based MCR. Besides, we comprehensively enhance the entire learning pipeline for aligning MCR spaces from the perspectives of training data, architecture, and learning objectives. With the preserved original modality alignment and the enhanced space alignment, Ex-MCR shows superior representation learning performance and excellent modality extensibility. To demonstrate the effectiveness of Ex-MCR, we align the MCR spaces of CLAP (audio-text) and ULIP (3D-vision) into the CLIP (vision-text), leveraging the overlapping text and image modality, respectively. Remarkably, without using any paired data, Ex-MCR learns a 3D-image-text-audio unified contrastive representation, and it achieves state-of-the-art performance on audio-visual, 3D-image, audio-text, visual-text retrieval, and 3D object classification tasks. More importantly, extensive qualitative results further demonstrate the emergent semantic alignment between the extended modalities (e.g., audio and 3D), which highlights the great potential of modality extensibility.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
