Set-CLIP：从低对齐多模态数据中通过分布视角探索对齐语义
Set-CLIP: Exploring Aligned Semantic From Low-Alignment Multimodal Data Through A Distribution View
AI总结：
本文提出Set-CLIP方法，将半监督多模态对齐视为流形匹配问题，通过语义密度分布损失从无配对数据提取隐式对齐，减少对严格匹配对的依赖，并在无配对监督数据时较CLIP提升144.83%。
摘要：
多模态融合打破了不同模态之间的边界，并已取得显著性能。然而，在许多专业领域，获取足够的对齐数据进行训练十分困难，这严重限制了以往有效模型的使用。因此，研究者尝试采用半监督学习方法，通过从匹配对较少的低对齐数据中学习来促进多模态对齐，但诸如伪标签等传统技术在标签缺乏的场景中可能会遇到困难。为了应对这些挑战，我们将半监督多模态对齐重新构建为一个流形匹配问题，并提出了一种基于CLIP的新方法，称为Set-CLIP。具体而言，通过设计一种新颖的语义密度分布损失，我们以细粒度约束潜在表示分布，并从无配对的多模态数据中提取隐式语义对齐，从而减少对大量严格匹配对的依赖。此外，我们应用粗粒度模态适应和单模态自监督指导，以缩小模态空间之间的差距并提高表示分布的稳定性。在包括蛋白质分析、遥感和通用视觉语言领域在内的各种领域的一系列任务上进行的广泛实验，验证了我们提出的Set-CLIP方法的有效性。特别是在没有配对数据进行监督训练的情况下，Set-CLIP依然表现出色，相比CLIP带来了144.83%的提升。
英文摘要：
Multimodal fusion breaks through the boundaries between diverse modalities and has already achieved notable performances. However, in many specialized fields, it is struggling to obtain sufficient alignment data for training, which seriously limits the use of previously effective models. Therefore, semi-supervised learning approaches are attempted to facilitate multimodal alignment by learning from low-alignment data with fewer matched pairs, but traditional techniques like pseudo-labeling may run into troubles in the label-deficient scenarios. To tackle these challenges, we reframe semi-supervised multimodal alignment as a manifold matching issue and propose a new methodology based on CLIP, termed Set-CLIP. Specifically, by designing a novel semantic density distribution loss, we constrain the latent representation distribution with fine granularity and extract implicit semantic alignment from unpaired multimodal data, thereby reducing the reliance on numerous strictly matched pairs. Furthermore, we apply coarse-grained modality adaptation and unimodal self-supervised guidance to narrow the gaps between modality spaces and improve the stability of representation distributions. Extensive experiments conducted on a range of tasks in various fields, including protein analysis, remote sensing, and the general vision-language field, validate the efficacy of our proposed Set-CLIP method. Especially with no paired data for supervised training, Set-CLIP is still outstanding, which brings an improvement of 144.83% over CLIP.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
