We propose Deep Multiset Canonical Correlation Analysis (dMCCA) as an extension to representation learning using CCA when the underlying signal is observed across multiple (more than two) modalities. We use deep learning framework to learn non-linear transformations from different modalities to a shared subspace such that the representations maximize the ratio of between- and within-modality covariance of the observations. Unlike linear discriminant analysis, we do not need class information to learn these representations, and we show that this model can be trained for complex data using mini-batches. Using synthetic data experiments, we show that dMCCA can effectively recover the common signal across the different modalities corrupted by multiplicative and additive noise. We also analyze the sensitivity of our model to recover the correlated components with respect to mini-batch size and dimension of the embeddings. Performance evaluation on noisy handwritten datasets shows that our model outperforms other CCA-based approaches and is comparable to deep neural network models trained end-to-end on this dataset.
Self-Supervised Multimodal Domino: in Search of Biomarkers for Alzheimer's Disease
Georgia Institute of Technology(佐治亚理工学院);Georgia State University(佐治亚州立大学);Emory University(埃默里大学);Center for Translational Research in Neuroimaging and Data Science(神经影像学与数据科学转化研究中心);Mila - Quebec AI Institute(米拉-魁北克人工智能研究所);Université de Montréal(蒙特利尔大学);Delft University of Technology(代尔夫特理工大学)
Maven: A Multimodal Foundation Model for Supernova Science
The NSF AI Institute for Artificial Intelligence and Fundamental Interactions(美国国家科学基金会人工智能与基础相互作用AI研究所);Department of Physics, Harvard University(哈佛大学物理系);Institute for Advanced Computational Science, Stony Brook University(石溪大学高级计算科学研究所);Massachusetts Institute of Technology(麻省理工学院);Center for Astrophysics — Harvard & Smithsonian(哈佛-史密森天体物理中心);Center for Theoretical Physics, Massachusetts Institute of Technology(麻省理工学院理论物理中心);Anthropic(Anthropic公司)
Multimodal Audio-Visual Information Fusion using Canonical-Correlated Graph Neural Network for Energy-Efficient Speech Enhancement
University of Wolverhampton(伍尔弗汉普顿大学);São Paulo State University(圣保罗州立大学);TECNALIA(泰克纳利亚研究中心);Basque Research & Technology Alliance(巴斯克研究与技术联盟);University of the Basque Country(巴斯克大学);Edinburgh Napier University(爱丁堡龙比亚大学)
A Mathematical Perspective On Contrastive Learning
Stores Foundational AI, Amazon, Palo Alto CA 94301 and Pasadena CA 91125(亚马逊公司、帕洛阿尔托加州94301和帕萨迪纳加州91125);Computing and Mathematical Sciences, California Institute of Technology, Pasadena CA 91125(计算与数学科学系,加州理工学院,帕萨迪纳加州91125)