对比关联：多模态生成模型的数据高效框架
Relating by Contrasting: A Data-efficient Framework for Multimodal Generative Models
摘要：
生成模型的多模态学习通常是指从多模态信息的共性中学习抽象概念，如视觉和语言。虽然它已被证明对学习可概括的表示是有效的，但此类模型的训练通常需要大量共享共性的“相关”多模态数据，这可能是昂贵的。为了缓解这一点，我们开发了一种新的生成模型学习的对比框架，使我们能够训练模型不仅是模态之间的共性，但“相关”和“不相关”的多模态数据之间的区别。我们在实验中表明，我们的方法可以在各种多模态VAE模型的具有挑战性的数据集上进行数据高效的多模态学习。我们还表明，在我们提出的框架下，生成模型可以准确地识别相关的样本无关的，使得它可以利用大量的未标记的，未配对的多模态数据。
英文摘要：
Multimodal learning for generative models often refers to the learning of abstract concepts from the commonality of information in multiple modalities, such as vision and language. While it has proven effective for learning generalisable representations, the training of such models often requires a large amount of "related" multimodal data that shares commonality, which can be expensive to come by. To mitigate this, we develop a novel contrastive framework for generative model learning, allowing us to train the model not just by the commonality between modalities, but by the distinction between "related" and "unrelated" multimodal data. We show in experiments that our method enables data-efficient multimodal learning on challenging datasets for various multimodal VAE models. We also show that under our proposed framework, the generative model can accurately identify related samples from unrelated ones, making it possible to make use of the plentiful unlabeled, unpaired multimodal data.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
