arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2306.04539cs.LGcs.CLcs.CVcs.ITmath.ITstat.ML
收藏

无需标注多模态数据的多模态学习:理论保证与应用

Multimodal Learning Without Labeled Multimodal Data: Guarantees and Applications

Carnegie Mellon University(卡内基梅隆大学);Columbia University(哥伦比亚大学);Princeton University(普林斯顿大学)

Paul Pu Liang, Chun Kai Ling, Yun Cheng, Alex Obolenskiy, Yudong Liu, Rohan Pandey, Alex Wilf, Louis-Philippe Morency, Ruslan Salakhutdinov

AI总结:

针对仅有标注单模态数据和未标注共现多模态数据的设定,论文提出多模态交互的信息论上下界,并验证其可用于性能估计、数据收集和模型选择。

摘要:

在许多联合利用多种模态进行学习的机器学习系统中,一个核心研究问题是理解多模态交互的本质:各模态如何结合,以提供任一单独模态中都不存在的、与任务相关的新信息。我们在一种半监督设定下研究这一交互量化挑战:该设定只有带标注的单模态数据以及自然共现的多模态数据(例如未标注图像与说明文字、视频及对应音频),而对这些多模态数据进行标注十分耗时。借助精确的信息论交互定义,我们的关键贡献是推导出下界与上界,用于量化该半监督设定中的多模态交互量。我们提出两个下界:一个基于模态之间的共享信息,另一个基于分别训练的单模态分类器之间的分歧;并通过与最小熵耦合近似算法的联系推导出一个上界。我们验证了这些估计边界,并展示它们如何准确追踪真实交互。最后,我们说明这些理论结果如何用于估计多模态模型性能、指导数据收集,以及为各类任务选择合适的多模态模型。

英文摘要:

In many machine learning systems that jointly learn from multiple modalities, a core research question is to understand the nature of multimodal interactions: how modalities combine to provide new task-relevant information that was not present in either alone. We study this challenge of interaction quantification in a semi-supervised setting with only labeled unimodal data and naturally co-occurring multimodal data (e.g., unlabeled images and captions, video and corresponding audio) but when labeling them is time-consuming. Using a precise information-theoretic definition of interactions, our key contribution is the derivation of lower and upper bounds to quantify the amount of multimodal interactions in this semi-supervised setting. We propose two lower bounds: one based on the shared information between modalities and the other based on disagreement between separately trained unimodal classifiers, and derive an upper bound through connections to approximate algorithms for min-entropy couplings. We validate these estimated bounds and show how they accurately track true interactions. Finally, we show how these theoretical results can be used to estimate multimodal model performance, guide data collection, and select appropriate multimodal models for various tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑