Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework
CMU(卡内基梅隆大学);Princeton University(普林斯顿大学);Columbia University(哥伦比亚大学);Harvard Medical School(哈佛医学院);Brigham and Women’s Hospital(布莱根妇女医院);University of Pennsylvania(宾夕法尼亚大学);University of Oregon(俄勒冈大学)
Paul Pu Liang, Yun Cheng, Xiang Fan, Chun Kai Ling, Suzanne Nie, Richard Chen, Zihao Deng, Nicholas Allen, Randy Auerbach, Faisal Mahmood, Ruslan Salakhutdinov,… 展开作者
Paul Pu Liang, Yun Cheng, Xiang Fan, Chun Kai Ling, Suzanne Nie, Richard Chen, Zihao Deng, Nicholas Allen, Randy Auerbach, Faisal Mahmood, Ruslan Salakhutdinov, Louis-Philippe Morency
The recent explosion of interest in multimodal applications has resulted in a wide selection of datasets and methods for representing and integrating information from different modalities. Despite these empirical advances, there remain fundamental research questions: How can we quantify the interactions that are necessary to solve a multimodal task? Subsequently, what are the most suitable multimodal models to capture these interactions? To answer these questions, we propose an information-theoretic approach to quantify the degree of redundancy, uniqueness, and synergy relating input modalities with an output task. We term these three measures as the PID statistics of a multimodal distribution (or PID for short), and introduce two new estimators for these PID statistics that scale to high-dimensional distributions. To validate PID estimation, we conduct extensive experiments on both synthetic datasets where the PID is known and on large-scale multimodal benchmarks where PID estimations are compared with human annotations. Finally, we demonstrate their usefulness in (1) quantifying interactions within multimodal datasets, (2) quantifying interactions captured by multimodal models, (3) principled approaches for model selection, and (4) three real-world case studies engaging with domain experts in pathology, mood prediction, and robotic perception where our framework helps to recommend strong multimodal models for each application.
A tutorial on discovering and quantifying the effect of latent causal sources of multimodal EHR data
Department of Biomedical Informatics Vanderbilt University Medical Center(生物医学信息学系范德堡大学医学中心);Department of Biomedical Informatics University of Pittsburgh(生物医学信息学系匹兹堡大学);Departments of Medicine & Biomedical Informatics Vanderbilt University Medical Center(医学与生物医学信息学系范德堡大学医学中心);Departments of Biomedical Informatics & Computer Science Vanderbilt University Medical Center & Vanderbilt University(生物医学信息学与计算机科学系范德堡大学医学中心及范德堡大学)
Visual Question Decomposition on Multimodal Large Language Models
Munich Center for Machine Learning(慕尼黑机器学习中心);Technical University of Munich(慕尼黑工业大学);Amazon Web Services(亚马逊云计算服务);LMU Munich(慕尼黑大学);MBZUAI(穆罕默德·本·扎耶德人工智能大学);University of Oxford(牛津大学)
Information-Theoretic Decomposition for Multimodal Interaction Learning
Gaoling School of Artificial Intelligence, Renmin University of China, Beijing(中国人民大学人工智能学院,北京);Beijing Key Laboratory of Research on Large Models(北京大模型研究关键实验室);Engineering Research Center of Next-Generation Intelligent Search(下一代智能搜索与推荐工程研究中心);Beihang University, Beijing(北航,北京);Gaotu Techedu Inc.(高图科技有限公司)
Explainable Multimodal Regression via Information Decomposition
Zhejiang University(浙江大学);Georgia Institute of Technology(佐治亚理工学院);Vrije Universiteit Amsterdam(阿姆斯特丹自由大学);UiT - The Arctic University of Norway(挪威北极大学)
Uncertainty-Aware Multimodal Learning via Conformal Shapley Intervals
Department of Statistics and Data Science, UCLA(统计与数据科学系, UCLA);Merck & Co(默克公司);Department of Biostatistics, UCLA(生物统计学系, UCLA);Departments of Statistics and Data Science, and of Biostatistics, UCLA(统计与数据科学系以及生物统计学系, UCLA)
Contrastive Learning on Multimodal Analysis of Electronic Health Records
Harvard T.H. Chan School of Public Health(哈佛T.H. 柏林公共卫生学院);Harvard Medical School(哈佛医学院);Rutgers University(罗格斯大学);National University of Singapore(新加坡国立大学)