In many machine learning systems that jointly learn from multiple modalities, a core research question is to understand the nature of multimodal interactions: how modalities combine to provide new task-relevant information that was not present in either alone. We study this challenge of interaction quantification in a semi-supervised setting with only labeled unimodal data and naturally co-occurring multimodal data (e.g., unlabeled images and captions, video and corresponding audio) but when labeling them is time-consuming. Using a precise information-theoretic definition of interactions, our key contribution is the derivation of lower and upper bounds to quantify the amount of multimodal interactions in this semi-supervised setting. We propose two lower bounds: one based on the shared information between modalities and the other based on disagreement between separately trained unimodal classifiers, and derive an upper bound through connections to approximate algorithms for min-entropy couplings. We validate these estimated bounds and show how they accurately track true interactions. Finally, we show how these theoretical results can be used to estimate multimodal model performance, guide data collection, and select appropriate multimodal models for various tasks.
Multimodal Deep Generative Model for Semi-Supervised Learning under Class Imbalance
Department of Industrial and Systems Engineering, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Republic of Korea(工业与系统工程系,韩国科学技术院(KAIST),大田,大韩民国)
Robult: Leveraging Redundancy and Modality Specific Features for Robust Multimodal Learning
Siebel School of Computing and Data Science, UIUC, US(UIUC计算机与数据科学学院);Department of Electrical and Computer Engineering, UIUC, US(UIUC电气与计算机工程学院);VinUni-Illinois Smart Health Center, VinUniversity, Vietnam(Vin大学-伊利诺伊智能健康中心)
Self-Training of Halfspaces with Generalization Guarantees under Massart Mislabeling Noise Model
Computer Science Laboratory (LIG)(计算机科学实验室(LIG));Department of Statistics (LJK)(统计系(LJK));Departement of Physics (SIMaP)(物理系(SIMaP));University of Grenoble Alpes(格勒诺布尔阿尔卑斯大学)
PI-MFM: Physics-informed multimodal foundation model for solving partial differential equations
Department of Statistics and Data Science, Yale University(统计与数据科学系,耶鲁大学);Department of Applied Mathematics and Statistics, Johns Hopkins University(应用数学与统计学系,约翰霍普金斯大学);Department of Applied Computational Mathematics and Statistics, University of Notre Dame(应用计算数学与统计学系,圣母大学);Department of Mathematics, University of California Los Angeles(数学系,加州大学洛杉矶分校);Department of Chemical and Environmental Engineering, Yale University(化学与环境工程系,耶鲁大学)
Cross-Modality Clustering-based Self-Labeling for Multimodal Data Classification
Faculty of Information;Communication Technology Wrocław University of Science;School of Computer Science University of Birmingham Birmingham, United Kingdom
Efficient Generalization via Multimodal Co-Training under Data Scarcity and Distribution Shift
Department of Electrical and Computer Engineering(电气与计算机工程系);Florida Institute for National Security (FINS)(佛罗里达国家安全研究所);Applied Artificial Intelligence Group(应用人工智能小组);University of Florida(佛罗里达大学)
MINT: Multimodal Instruction Tuning with Multimodal Interaction Grouping
University of California, San Diego(加州大学圣地亚哥分校);Johns Hopkins University(约翰霍普金斯大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Massachusetts Institute of Technology(麻省理工学院)
本文提出 MINT 任务分组策略,按多模态交互类型组织指令微调任务,以学习可迁移技能并减少任务干扰,在泛化与专业化间取得更优平衡。
Causal Representation Learning from Multimodal Biomedical Observations
Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Carnegie Mellon University(卡内基梅隆大学);University of Bristol(布里斯托大学);Broad Institute of MIT and Harvard(麻省理工学院与哈佛大学博德研究所)
MMBind: Unleashing the Potential of Distributed and Heterogeneous Data for Multimodal Learning in IoT
Hong Kong University of Science and Technology(香港科技大学);University of California, Los Angeles(加州大学洛杉矶分校);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);U.S. Army DEVCOM Army Research Laboratory(美国陆军DEVCOM陆军研究实验室)
MUDI: A Multimodal Biomedical Dataset for Understanding Pharmacodynamic Drug-Drug Interactions
VNU University of Engineering and Technology (VNU-UET)(越南工程技术大学);Lausanne University Hospital (CHUV)(洛桑大学医院);University of Lausanne (UNIL)(洛桑大学);Hanoi Medical University(河内医学院);National Geriatric Hospital, Hanoi(河内国立老年医院)
Weakly-supervised anomaly detection for multimodal data distributions
School of Marine Science and Technology(海洋科学与技术学院);Northwestern Polytechnical University(西北工业大学);School of Computing(计算机学院);University of Eastern Finland(东芬兰大学);University of Turku(图尔库大学);Singapore Institute of Technology(新加坡理工学院)
Enhancing ensemble learning and transfer learning in multimodal data analysis by adaptive dimensionality reduction
UiT The Arctic University of Norway(挪威北极圈大学);University of Cambridge(剑桥大学);The Alan Turing Institute(艾伦·图灵研究所);University of Grenoble Alpes(格勒诺布尔阿尔卑斯大学);Institut Universitaire de France(法国大学研究院)