arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2302.12247cs.LGcs.AIcs.CLcs.CVcs.ITmath.IT
收藏

量化与建模多模态交互:一种信息分解框架

Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework

CMU(卡内基梅隆大学);Princeton University(普林斯顿大学);Columbia University(哥伦比亚大学);Harvard Medical School(哈佛医学院);Brigham and Women’s Hospital(布莱根妇女医院);University of Pennsylvania(宾夕法尼亚大学);University of Oregon(俄勒冈大学)
Paul Pu Liang, Yun Cheng, Xiang Fan, Chun Kai Ling, Suzanne Nie, Richard Chen, Zihao Deng, Nicholas Allen, Randy Auerbach, Faisal Mahmood, Ruslan Salakhutdinov,… 展开作者

Paul Pu Liang, Yun Cheng, Xiang Fan, Chun Kai Ling, Suzanne Nie, Richard Chen, Zihao Deng, Nicholas Allen, Randy Auerbach, Faisal Mahmood, Ruslan Salakhutdinov, Louis-Philippe Morency

AI总结:

本文提出基于信息论的 PID 框架,量化多模态输入与任务间的冗余、独特和协同信息,并通过可扩展估计器、基准实验及真实案例支持数据集分析、模型交互度量与模型选择。

摘要:

近期人们对多模态应用的兴趣爆发式增长,催生了大量用于表示和整合来自不同模态信息的数据集与方法。尽管取得了这些实证进展,仍存在一些基础性研究问题:我们如何量化解决一项多模态任务所必需的交互?进而,哪些多模态模型最适合捕捉这些交互?为回答这些问题,我们提出一种信息论方法,用于量化输入模态与输出任务之间相关的冗余、独特和协同程度。我们将这三种度量称为多模态分布的 PID 统计量(简称 PID),并引入两种新的 PID 统计量估计器,它们可扩展到高维分布。为验证 PID 估计,我们在 PID 已知的合成数据集以及大规模多模态基准上开展了大量实验,在后者中将 PID 估计与人工标注进行比较。最后,我们展示了其在以下方面的用途:(1)量化多模态数据集内的交互,(2)量化多模态模型捕捉到的交互,(3)有原则的模型选择方法,以及(4)三个真实世界案例研究,涉及病理学、情绪预测和机器人感知领域的领域专家;在这些应用中,我们的框架有助于为每个应用推荐性能强劲的多模态模型。

英文摘要:

The recent explosion of interest in multimodal applications has resulted in a wide selection of datasets and methods for representing and integrating information from different modalities. Despite these empirical advances, there remain fundamental research questions: How can we quantify the interactions that are necessary to solve a multimodal task? Subsequently, what are the most suitable multimodal models to capture these interactions? To answer these questions, we propose an information-theoretic approach to quantify the degree of redundancy, uniqueness, and synergy relating input modalities with an output task. We term these three measures as the PID statistics of a multimodal distribution (or PID for short), and introduce two new estimators for these PID statistics that scale to high-dimensional distributions. To validate PID estimation, we conduct extensive experiments on both synthetic datasets where the PID is known and on large-scale multimodal benchmarks where PID estimations are compared with human annotations. Finally, we demonstrate their usefulness in (1) quantifying interactions within multimodal datasets, (2) quantifying interactions captured by multimodal models, (3) principled approaches for model selection, and (4) three real-world case studies engaging with domain experts in pathology, mood prediction, and robotic perception where our framework helps to recommend strong multimodal models for each application.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑