多模态与单模态对比学习的比较研究
On the Comparison between Multi-modal and Single-modal Contrastive Learning
AI总结：
本研究提出特征学习理论框架，对比多模态与单模态对比学习，发现信噪比是关键因素，多模态通过模态协同实现更优特征学习与下游性能提升。
摘要：
基于语言监督的多模态对比学习已引领现代机器学习领域的范式转变。通过在网络规模的数据集上进行预训练，多模态对比学习能够学习到高质量的表征，展现出卓越的鲁棒性和可迁移性。尽管其经验上取得了成功，但其理论理解仍处于起步阶段，尤其是在与单模态对比学习的比较方面。在本工作中，我们引入了一个特征学习理论框架，为理解多模态与单模态对比学习之间的差异提供理论基础。基于由信号和噪声组成的数据生成模型，我们的分析在采用InfoMax目标函数训练的ReLU网络上进行。通过基于轨迹的优化分析以及在下游任务上的泛化表征，我们识别出影响多模态和单模态对比学习在下游任务中泛化能力的关键因素，即信噪比（SNR）。通过两种模态之间的协同合作，多模态学习能够实现更好的特征学习，从而相比单模态学习在下游任务中带来性能提升。我们的分析提供了一个统一框架，能够刻画单模态和多模态对比学习的优化与泛化过程。在合成数据集和真实数据集上的实证实验进一步巩固了我们的理论发现。
英文摘要：
Multi-modal contrastive learning with language supervision has presented a paradigm shift in modern machine learning. By pre-training on a web-scale dataset, multi-modal contrastive learning can learn high-quality representations that exhibit impressive robustness and transferability. Despite its empirical success, the theoretical understanding is still in its infancy, especially regarding its comparison with single-modal contrastive learning. In this work, we introduce a feature learning theory framework that provides a theoretical foundation for understanding the differences between multi-modal and single-modal contrastive learning. Based on a data generation model consisting of signal and noise, our analysis is performed on a ReLU network trained with the InfoMax objective function. Through a trajectory-based optimization analysis and generalization characterization on downstream tasks, we identify the critical factor, which is the signal-to-noise ratio (SNR), that impacts the generalizability in downstream tasks of both multi-modal and single-modal contrastive learning. Through the cooperation between the two modalities, multi-modal learning can achieve better feature learning, leading to improvements in performance in downstream tasks compared to single-modal learning. Our analysis provides a unified framework that can characterize the optimization and generalization of both single-modal and multi-modal contrastive learning. Empirical experiments on both synthetic and real-world datasets further consolidate our theoretical findings.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
