To gain insight into the mechanisms behind machine learning methods, it is crucial to establish connections among the features describing data points. However, these correlations often exhibit a high-dimensional and strongly nonlinear nature, which makes them challenging to detect using standard methods. This paper exploits the entanglement between intrinsic dimensionality and correlation to propose a metric that quantifies the (potentially nonlinear) correlation between high-dimensional manifolds. We first validate our method on synthetic data in controlled environments, showcasing its advantages and drawbacks compared to existing techniques. Subsequently, we extend our analysis to large-scale applications in neural network representations. Specifically, we focus on latent representations of multimodal data, uncovering clear correlations between paired visual and textual embeddings, whereas existing methods struggle significantly in detecting similarity. Our results indicate the presence of highly nonlinear correlation patterns between latent manifolds.
Uncovering Trajectory and Topological Signatures in Multimodal Pediatric Sleep Embeddings
Department of Radiology(放射科);University of California San Francisco(加州大学旧金山分校);UNC–Chapel Hill(北卡罗来纳大学教堂山分校);UNC School of Data Science and Society(北卡罗来纳大学教堂山分校数据科学与社会学院)
Hidden in the Multiplicative Interaction: Uncovering Fragility in Multimodal Contrastive Learning
Berlin Institute of Health, Charité - Universitätsmedizin Berlin(柏林健康研究所,柏林查理医院);Intelligent Medicine Institute, Fudan University(智能医学研究院,复旦大学);Department of Mathematics and Computer Science, Freie Universität Berlin(数学与计算机科学系,柏林自由大学)
Multimodal Inverse Attention Network with Intrinsic Discriminant Feature Exploitation for Fake News Detection
School of Information Science and Engineering, Shandong Normal University(山东师范大学信息科学与工程学院);Faculty of Engineering & Information Technology, University of Technology Sydney(悉尼科技大学工程与信息技术学院)
Uncovering spatial tissue domains and cell types in spatial omics through cross-scale profiling of cellular and genomic interactions
Institute for Computational and Mathematical Engineering, Stanford University(计算与数学工程研究所,斯坦福大学);Department of Radiation Oncology, Stanford University(放射肿瘤学系,斯坦福大学);Gladstone Institute of Cardiovascular Disease, San Francisco, CA(心血管疾病 Gladstone 研究所,旧金山,CA)
ContIG: Self-supervised Multimodal Contrastive Learning for Medical Imaging with Genetics
Hasso Plattner Institute for Digital Engineering, University of Potsdam(波茨坦大学哈索·普拉特纳数字工程研究所);TU Kaiserslautern(凯泽斯劳滕工业大学);Hasso Plattner Institute for Digital Health at the Icahn School of Medicine at Mount Sinai(西奈山伊坎医学院哈索·普拉特纳数字健康研究所)
Revealing Multimodal Causality with Large Language Models
University of Technology Sydney(技术大学悉尼大学);Tongji University(同济大学);University of Melbourne(墨尔本大学);University of Sydney(悉尼大学);Macquarie University(麦考瑞大学)
MM-SpuBench: Towards Better Understanding of Spurious Biases in Multimodal LLMs
University of Virginia(弗吉尼亚大学);Purdue University(普渡大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Georgia Institute of Technology(佐治亚理工学院)
Harmonized Multimodal Learning with Gaussian Process Latent Variable Models
University of Chinese Academy of Sciences(中国科学院大学);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);Noah’s Ark Lab(诺亚方舟实验室);The University of Texas at San Antonio(德克萨斯大学圣安东尼奥分校)
Contrastive Learning on Multimodal Analysis of Electronic Health Records
Harvard T.H. Chan School of Public Health(哈佛T.H. 柏林公共卫生学院);Harvard Medical School(哈佛医学院);Rutgers University(罗格斯大学);National University of Singapore(新加坡国立大学)